Create a simple way to run this on local doc files

jenweber commented 7 years ago

I was able to get it to run on local files after bypassing a lot of things by commenting them out. It would be good to have it as an options flag. I don't think it would be too hard, since there's already an option flag for individual versions. I'm wondering if I just named things perfectly and put them in place if it would work as-is.

jenweber commented 7 years ago

My working approach from start to finish

Clone and open Ember.js core
Generate YUIdocs by running yuidoc . which creates a docs/ directory of files. The important file is data.json, which contains all the documentation and internal relationships
Clone and open ember-jsonapi-docs
From the command line, export some AWS keys as described in the readme (I found that some scoped IAM keys I had were insufficient, something something permissions)
In the same console tab, run yarn start -- --project ember --version 2.11.0 or similar. Just pick a version. This will download some stuff from s3, generate a big filetree under the tmp/ directory, and process the specified version to turn it into JSONAPI. You'll use that tree as the skeleton for operating on your local yuidoc files.
Grab that data.json file that was generated in step 2. Paste it into tmp/s3-docs/v2.11.0

Rename the v2.11.0 directory to a bogus version that doesn't exist, like 2.11.18. Whatever you rename it to, change the version in data.json to match:

{
"project": {
    "name": "The Ember API",
    "description": "The Ember API: a framework for building ambitious web applications",
    "url": "https://emberjs.com/",
    "version": "2.11.18" <<<<<<<<< Change version
},

Delete tmp/v2.11.0/ember-docs.json. Rename data.json to ember-docs.json
Run the jsonapi generator again, but specify the made up version yarn start -- --project ember --version 2.11.18
It will look like things are downloading. Your json file from yuidocs will be transformed into jsonapi. You should see the results in tmp/rev-index/ember-2.11.18.json and tmp/json-docs/ember/2.11.8
Copy json-docs and rev-index into the public directory of ember-api-docs.
The ember-api-docs will look at two files to see which versions of the ember documentation are available, rev-index/ember-data.json and rev-index/ember.json. However you only generated one version's worth of jsonapi docs. So, to convince the app that your 2.11.18 version is the only one that exists, remove mentions of all other versions from those two files, ember-data.json and ember.json. The result should look similar to this:

{
  "data": {
    "id": "ember",
    "type": "project",
    "attributes": {
      "github-url": "https://github.com/emberjs/ember.js"
    },
    "relationships": {
      "project-versions": {
        "data": [{
          "id": "ember-2.11.18",
          "type": "project-version"
        }]
      }
    }
  },
  "meta": {
    "availableVersions": ["2.11.18"]
  }
}

Change the adapter to point to itself (your files in public) instead of an API endpoint by changing the host in app/adapters/application.js to host: 'http://localhost:4200'
ember serve and hopefully you'll see the app up and running

If links are broken for the modules, you can temporarily comment out these lines in app/routes/project-version/modules/module.js to get them working, unless your modules names include slashes. The app expects that all modules will be named like ember-something so simple module names like component will fail in the routing without this fix

getModel(typeName, params, transition) {
const projectID = transition.params['project-version'].project;
const compactVersion = transition.params['project-version'].project_version;
const projectVersion = this.get('metaStore').getFullVersion(projectID, compactVersion);
let klass = params[typeName];
// WHAT DOES THIS DO????
// if (!~klass.indexOf(projectID)) {
//   klass = `${projectID}-${klass}`;
// }
return this.find(typeName, `${projectID}-${projectVersion}-${klass}`);
},

Debug log, because why not. Don't follow these:

This story has a happy ending. Here are my debugging steps, which may aid in coming up with a streamlined set of instructions:

Open ember.js and run yuidoc . which generates a folder called docs which contains a bunch of html files, but the important one is data.json

In ember-jsonapi-docs, comment out a few things:

// downloadExistingDocsToLocal()
//     .then(() => fetchYuiDocs(projects, specificDocsVersion))
//     .then(() =>
    readDocs(projects, specificDocsVersion)
...
// .then(uploadToS3)

Based on errors thrown, it seems to expect a filetree that looks like this, so I made that structure: tmp/json-docs/ember/projects tmp/json-docs/ember-data/projects tmp/rev-index/

I dropped data.json into tmp, and renamed the version stated in data.json to 2.16.0 because I am suspicious of the alpha.

Errors indicate a missing file at tmp/json-docs/ember/projects/ember.json so I rename data.json to ember.json and put it at that location. Subsequent error of "cannot read relationships of undefined" indicates that a different file is needed. Sub in a file I got from Todd, trimmed down to only have 2.16.0. Name it ember.json. My YUIdoc output gets renamed back to data.json

Contents of ember.json

{
  "data": {
    "id": "ember",
    "type": "project",
    "attributes": {
      "github-url": "https://github.com/emberjs/ember.js"
    },
    "relationships": {
      "project-versions": {
        "data": [{
          "id": "ember-2.16.0",
          "type": "project-version"
        }]
      }
    }
  },
  "meta": {
    "availableVersions": ["2.16.0"]
  }
}

After yarn start, nothing happens except:

Downloading docs for ember & ember-data
Reading project files
Done in 1.21s.

It seems that download & fetch are not skippable. The app needs all those byproducts in order to function.

So I added some AWS keys via export in the command line and put the download & fetch back, to see if I can just download something else and run that. (Run into AWS access problems, try a different key, etc)

Run yarn start -- --project ember --version 2.11.0

Lots of files get created. Based on the filetree, see that I should have put my docs into a folder called tmp/s3-docs/v2.16.0/ember-docs.json, maybe??? Decide to just roll with 2.11.0 to see if it works.

Based on past work, I know that CORS is an issue, so I run this from just inside tmp/

http-server -p 8000 --cors

Open up ember-api-docs. Change the adapter host in app/adapters/application.js to host: 'http://localhost:8000'

Start the ember-api-docs app with ember s and open it in the browser. Error displayed in the DOM:

FetchError: invalid json response body at http://localhost:8000/rev-index/ember.json reason: Unexpected end of JSON input

Back end server confirms a failed GET for /rev-index/ember.json but also indicates maybe some sort of passed preflight check??? The file is there in the tree...

[Sat Sep 02 2017 19:49:01 GMT-0400 (EDT)] "GET /rev-index/ember.json" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)"
[Sat Sep 02 2017 19:49:01 GMT-0400 (EDT)] "GET /rev-index/ember.json" Error (404): "Not found"

Check the network requests.

FetchError: invalid json response body at http://localhost:8000/rev-index/ember.json

Become suspicious of json. Remove trailing whitespace. No change in errors.

Become suspicious of the http server. Dump the files into public. Set the adapter host to host: '/'. Get this error: TypeError: Only absolute URLs are supported

Delete host altogether. Same error. Try setting host to http://locahost:4200. See a new error: FetchError: invalid json response body at http://localhost:4200/rev-index/ember-2.15.0.json reason: Unexpected token < in JSON at position 0

Follow that path in the app and see that it doesn't exist, because I ran the doc generator for 2.11.0. Try hard coding that into the adapter:

  currentProject: '2.11.0',
  currentProjectVersion: '2.11.0',

Same error. Search the codebase for 2.15. Remove everything but 2.11 from the availableVersions array in rev-index/ember.json. Get a new error:

FetchError: invalid json response body at http://localhost:4200/rev-index/ember-undefined.json

This file doesn't exist. Go digging through Todd's files. It's not there either. Realize that there probably shouldn't be a file named "ember-undefined" and that's probably a bad filename parsing error. Go back to ember.json and ember-data.json and remove mentions of anything but 2.11. Regret not running the build of everything. Restart server just in case. Consider throwing my laptop out the window while I wait for it to build.

FINALLY it works.

Generate YUIdocs with modules changes, drop it into the 2.11.0 files. Get this error:

ember-2.11.0 has already been indexed in json-docs

Give it a bogus version name 2.11.18 as the folder name and version stated in the json. Change the ember.json and ember-data.json in ember-api-docs to reflect this. Drop the file into the place that 2.11 was. Restart ember server. Hard code 2.11.18 into the adapter. Hit a Fastboot error....

TypeError: Cannot read property 'split' of undefined

More than 2.11 json was needed. Go back to YUIdocs. Run generate with the bogus version name, and all of index.js intact in jsonapi-docs (except S3 upload).

yarn start -- --project ember --version 2.11.18

It will look like it is downloading things. Copy ALL the files over into public. Don't forget to limit the scope of ember.json and ember-data.json.

toddjordan commented 7 years ago

Great instructions. Followed and things working nicely. I think we can put this in a single, local script.

copy over docs build output (using the generated version in the doc file as the tmp/s3-docs version directory)
run the ember-jsonapi-docs build on the specified version
copy output into ember-api-docs repo (assuming or supplying the repo location)

Additionally, we should just add an env variable, like USE_LOCAL_DATA that will set API_HOST to http://localhost:4200

would be cool to have a developer just do something like (assuming all repos cloned into the same parent directory):

cd ember.js
npm run docs
cd ../ember-jsonapi-docs
npm run generate-local-docs
cd ../ember-api-docs/
USE_LOCAL_DATA=true ember s

cc @sivakumar-kailasam @jenweber

toddjordan commented 7 years ago

I have a script now that does some of this, but needs to be cleaned up. stay tuned...

ember-learn / ember-jsonapi-docs

Create a simple way to run this on local doc files #35

My working approach from start to finish

Debug log, because why not. Don't follow these: