Rejigged README a little

[philsturgeon]

I'm excited about API Flow, but the README had me installing from source with me only noticing I could just yarn install later. I think the default for most folks is installing the package, not building from source.

I've also added const path = require('path'); as it was erring before that.

Still, this is giving me trouble:

$ node scripts/openapi_to_postman.js
/Users/psturgeon/src/apis/scripts/openapi_to_postman.js:15
const promise = ApiFlow.transform({
                        ^

TypeError: ApiFlow.transform is not a function
    at Object.<anonymous> (/Users/psturgeon/src/apis/scripts/openapi_to_postman.js:15:25)
    at Module._compile (module.js:569:30)
    at Object.Module._extensions..js (module.js:580:10)
    at Module.load (module.js:503:32)
    at tryModuleLoad (module.js:466:12)
    at Function.Module._load (module.js:458:3)
    at Function.Module.runMain (module.js:605:10)
    at startup (bootstrap_node.js:158:16)
    at bootstrap_node.js:575:3

I'd love to get the README in line with how things work, just gimme some hints! :)

[MikeRalphson]

@philsturgeon I believe the import statement needs to be:

const ApiFlow = require('api-flow').default; // if from npm
const ApiFlow = require('./dist/node/api-flow.js').default; // if from `make runners TARGET="node"`

i.e. add the .default property.

At this point, I'm able to do a transform - notwithstanding some issues with AMD modules (know-your-http-well not being webpacked correctly on Windows) and the result of the path.resolve above not producing valid URI's on Windows.

[philsturgeon]

Thank you for replying @MikeRalphson!

My debugging got me that far, I just get a different error:

const promise = ApiFlow.transform({
                        ^

TypeError: ApiFlow.transform is not a function
    at Object.<anonymous> (/Users/psturgeon/src/apis/scripts/openapi_to_postman.js:15:25)
    at Module._compile (module.js:569:30)
    at Object.Module._extensions..js (module.js:580:10)
    at Module.load (module.js:503:32)
    at tryModuleLoad (module.js:466:12)
    at Function.Module._load (module.js:458:3)
    at Function.Module.runMain (module.js:605:10)
    at startup (bootstrap_node.js:158:16)
    at bootstrap_node.js:575:3

Content of that script file is:

const ApiFlow = require('api-flow').default;
const path = require('path');

const options = {
  source: {
      format: 'swagger',
      version: 'v2.0'
  },
  target: {
      format: 'postman',
      version: 'v2.0'
  }
}

const promise = ApiFlow.transform({
  options,
  uri: path.resolve(__dirname, './tmp/interstellar/docs/api.yml')
})

promise.then((data) => {
  debugger;
})

[MikeRalphson]

@philsturgeon did you use the package from npm, or build the code yourself as per this comment? Your script works fine / gets past that point here with the copy of API-Flow I've built from source.

You could try adding

const util = require('util')
console.log(util.inspect(ApiFlow))

after the require of ApiFlow to see what's going on.

[philsturgeon]

@MikeRalphson doh! Ok, I missed that reply from @JonathanMontane. I don't want that thread turning into a support one, but here is appropes for that.

I skipped the nvm use step as I am not using nvm and dont know what that's all about.

When I use the example code from Jonathans comment, I get:

/Users/psturgeon/src/apis/dist/node/api-flow.js:37672
self.onmessage = function () {
^

ReferenceError: self is not defined
    at Object.<anonymous> (/Users/psturgeon/src/apis/dist/node/api-flow.js:37672:1)
    at __webpack_require__ (/Users/psturgeon/src/apis/dist/node/api-flow.js:30:30)
    at /Users/psturgeon/src/apis/dist/node/api-flow.js:76:18
    at /Users/psturgeon/src/apis/dist/node/api-flow.js:79:10
    at webpackUniversalModuleDefinition (/Users/psturgeon/src/apis/dist/node/api-flow.js:3:20)
    at Object.<anonymous> (/Users/psturgeon/src/apis/dist/node/api-flow.js:10:3)
    at Module._compile (module.js:569:30)
    at Object.Module._extensions..js (module.js:580:10)
    at Module.load (module.js:503:32)
    at tryModuleLoad (module.js:466:12)

[MikeRalphson]

@philsturgeon don't you just love webpack? There's no way of matching up line numbers between our builds unless we have the exact same order of directories in our node_modules. I don't have any self.onmessage = in my dist/node/api-flow.js at all though, so I suspect it's something webpack is adding. What node.js version are you using?

Adding --json to the webpack commandline will output some debugging information to help trace which module id comes from where.

[philsturgeon]

I was running v8.0.0 and now I'm on v7.4.0 via nodenv. My yarn was 0.24.1 and now it's v1.1.0.

Rebuilt everything and I'm getting the same error about self.onmessage.

I tried adding --json to http://github.com/WeConnect/apis/blob/3f327d661b00a39ef0e2d5d5b4f7e22de8a3fc23/API-Flow/scripts/runners.sh#L12 and it's just barfing JavaScript all over the terminal output when I run make again.

[philsturgeon]

Commenting out the offending self.onmessage line, I get a bunch of errors about XMLHttpRequest not being there:

node ./scripts/openapi_to_postman.js
(node:34311) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 3): ReferenceError: XMLHttpRequest is not defined
(node:34311) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

What the heck is going on :D

[JonathanMontane]

@philsturgeon Your issue mystified for a long while until I realized there's a dumb error in the webpack.config file of the webworker.

change

  ...
  output: {
    path: path.resolve(__dirname, '../../../dist/node/'),
    filename: 'api-flow.js',
    libraryTarget: 'umd'
  },
  ...

to

  ...
  output: {
    path: path.resolve(__dirname, '../../../dist/webworker/'),
    filename: 'api-flow.js',
    libraryTarget: 'umd'
  },
  ...

alternatively, just use make runners TARGET="node" until this bug is fixed.

[MikeRalphson]

That makes sense, as I was building for node only, a) for speed and b) cos I don't have make on this platform (Windows with git-for-windows / MINGW).

[philsturgeon]

Thank you for all the help! I did multiple targets in the first place because I wasn't sure which I needed (another point I'll add to the README). Since then I'd been doing just make runners TARGET="node".

Deleting dist/ and running just make runners TARGET="node" seems to have got me past the ReferenceError: self is not defined issue, but I am still getting the UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 3): ReferenceError: XMLHttpRequest is not defined issue.

UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 3): ReferenceError: XMLHttpRequest is not defined
(node:22093) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

I'm just using your example @JonathanMontane so I'm not sure what is causing this.

[philsturgeon]

@JonathanMontane hey did you see my reply? Life is still terrible! :)

[JonathanMontane]

@philsturgeon sorry for the late reply, I've been swamped. I am a bit stumped. Can you try adding a .catch((e) => console.error(e)) to the transform.then(), this will hopefully give us more information about the line that's failing.

[philsturgeon]

    at /Users/psturgeon/src/apis/dist/node/api-flow.js:11559:15
    at Promise (<anonymous>)
    at F (/Users/psturgeon/src/apis/dist/node/api-flow.js:5358:36)
    at Object.methods.httpResolve [as resolve] (/Users/psturgeon/src/apis/dist/node/api-flow.js:11558:10)
    at Object.methods.resolve (/Users/psturgeon/src/apis/dist/node/api-flow.js:11975:31)
    at Object.methods.load (/Users/psturgeon/src/apis/dist/node/api-flow.js:12063:32)
    at Function.load (/Users/psturgeon/src/apis/dist/node/api-flow.js:11867:22)
    at Object.methods.load (/Users/psturgeon/src/apis/dist/node/api-flow.js:11132:17)
    at Object.methods.transform (/Users/psturgeon/src/apis/dist/node/api-flow.js:11343:18)
    at Function.transform (/Users/psturgeon/src/apis/dist/node/api-flow.js:11046:22)```

[JonathanMontane]

@philsturgeon Alright this is weird, but hopefully, we can solve this.

It seems that the environment that is used is the web environment instead of the node environment for some reason. Can you check your api-flow-config.js file in the configs/node folder, and see that the first line is import Environment from '../../src/environments/node/Environment'? If it's not, that would be your issue and would be an easy fix.

If the file match, then maybe there's an issue with your node environment file directly. In that case, can you check that the file in src/environments/node/ does not use XMLHttpRequest anywhere (it normally shouldn't)?

If that's not the case, then I suppose the issue is with the Webpack resolver that fails for some reason, and then I'm clueless.

[philsturgeon]

You got it on the first throw!

I'm seeing this in api-flow-config:

import Environment from '../../src/environments/web/Environment'

Here's everything I did to get there, deleting everything and starting again:

rm -rf API-Flow
git clone https://github.com/luckymarmot/API-Flow.git
cd API-Flow
node --version # 8.6.0
yarn install
make install # doesnt exist oops, will add to PR
make
make runners TARGET="node"
make transfer TARGET="swagger postman2"

In previous iterations I think I'd done yarn and make in the wrong order and got a bunch of warnings, but this time I got it all 100% right aaaaand I'm still getting '../../src/environments/web/Environment` all up in there.

Thanks for helping me with this!!

[JonathanMontane]

@philsturgeon Glad to see it finally worked out for you. Sorry that it took so long to fix though.

[philsturgeon]

Ahh well no it's not resolved, I changed that line of code but I'm struggling to rebuild. I ran make and make runners... again but the same error persisted.

Next step was deleting ../dist/node/api-flow.js and running make plus make runners... again but I've not got a new ../dist/node/api-flow.js file.

I'd like to document the flow for a full rebuild, as I have no idea what to do now. Tried make clean and repeating but no dice.

I deleted everything and started again and still ../dist/ is not being created.

[philsturgeon]

HUZZAH!

Ok, for some reason a previous attempts had been placing the dist folder in foo/dist instead of foo/API-Flow/dist.

Now my script has:

const ApiFlow = require('../API-Flow/dist/node/api-flow.js').default;

ALL GOOD!

Updating my PR with instructions from my history so new folks don't get as lost and sad. Please commit anything you discovered as bugs during this thread.

[philsturgeon]

Actually now I'm getting Error: could not convert Api(s): missing target format as im seeing a mixture of postman postman2 and postman-collection in comments and READMEs. I'm adding debug and running make again and the debug isn't showing. Literally replacing that error with new Error('SHARKS');and getting nothing.

I don't understand this workflow. I want to learn it and document it, but... 🤷‍♂️

[JonathanMontane]

@philsturgeon I don't have the hand on this repository anymore, so I am as stuck as you when it comes to updating it (you can see my fork for a more frequently updated version). However, you're on the right track here, and you just have to add a couple of lines to one files and you'll be good to go.

Here's a bit of background knowledge so that you understand how everything works and you'll see that it's not that complex, it's just that this library had to work in many conflicting environments in a quite modular way.

A Quick rundown of how API-Flow works

There are 5 core concepts:

• Environments
• Loaders
• Parsers
• Internal Model
• Serializers

The Environments give methods to handle I/O ops that are used by the Loaders to fetch the original resource and all its associated resources, which are normalized and then passed to the Parsers, which convert the data into the Internal Model, which in turn is transformed into an output format by Serializers.

Environments

These files contain wrappers for I/O operations in the different environments as a unified interface so that other parts of the library can forget about where they are run. This allows the library to be run in Node, the web, in web workers, and even JavascriptCore.

You can find these files under src/environments/*.

Loaders

These files contain small methods that fetch the original resource from its uri, analyzes it and fetches all the other resources that this file may require. This is also the place where you can reduce the variability in your input, so that you have an easier time writing your parser afterwards. For example, swagger 2 can have implicit hostnames based on where the file is located. Implicit information generally doesn't translate well in other formats, so we make the host explicit at this stage. Another example would be the postman collection v2 format, where you can basically write a uri in about 10 different ways. Handling all these cases every time you encounter a URI while parsing the file would be a pain, and a good way to add errors to your code, so we normalize the URI scheme at this step.

You can find these files under src/loaders/*

Parsers, Internal Model and Serializers

These are fairly easy to understand to be honest. You can find them all in their respective folders src/parsers/*, src/models/*, src/serializers/*

How everything is bound together

Once you have all these bits that exist, you need to bind them together. This is where the core library (found in src/api-flow.js) comes in. It exposes a simple interface (setup, load, parse, serialize, and a transform method, which is just a convenience method for load -> parse -> serialize).

However this core file needs to load the correct dependencies depending on the environment it is running in. For instance, for the web, it needs the src/environments/web/Environment file, but it also needs the web-ready version of the raml-1-parser (which was not an easy thing to get to compile at the time).

So in order to have as much liberty as possible while avoiding duplicating code, I chose to create simple config files, that are just imports/exports of all the specific files that you want in that specific setup.

These files can be found in the configs/* folders under the name api-flow-config.js. And if you have a look at one, you'll see that they are extremely trivial.

Why Postman isn't working in Node (and how to resolve it)

Now for why you don't seem to be able to convert from or to Postman collections in Node: It's quite simple, if you look at the api-flow-config.js for the node setup (i.e. configs/node/api-flow-config.js) you'll see that it only has the Swagger and the RAML loaders/parsers/serializers included in it. If you add the Postman loader, parser and serializer, and build the lib it should now work with Postman.

As for why Postman isn't included, it's pretty simple. I forgot to add it. This library is mainly used inside Paw, rather than Node and the Postman format is one of the last formats I added to the library, so I forgot to add it to all the environments. I apologize for the inconvenience.

Finally, Once you feel like you've got the hang of how everything is arranged, please feel free to create another config that suits exactly your needs (for instance that includes only postman and swagger, which can drastically reduce bundle size).

What format name to use

you said you found postman, postman2, and postman-collection and weren't sure which was the correct one to use. It's postman-collection for the input, and postman for the output (I've created an issue to fix this inconsistency, it will be postman-collection all the time in the future). If you have doubts about the name to use for a certain parser or serializer, you can check the __meta__ attribute at the beginning of the corresponding parser.

I definitely agree that there's some effort to be made about the onboarding process though. Thank you for your patience anyway.

[philsturgeon]

Hey @JonathanMontane! You are awesome, this reply is fantastic, and I'm really grateful. I'm working with your fork as clearly theres a bunch of fixes in there, and that's a great place to start.

Your outline of the arch of this app is amazing. I've created a PR to your repo with some fixes I needed to get things working, and now everything works.

I'll keep playing around with your fork and try to get my solution to production, and when its done I'll update the README PR with some tips to newcomers on how stuff actually works.

[philsturgeon]

Seeing as this thread turned into a Phil Sturgeon Support Thread, I'm closing this for the more concise #147. Thanks to everyone that helped. Beers on me in NYC. 👍