Where to publish a directory of APIs?

[MikeTheCanuck]

In 2017 season there was an effort (incomplete) to publish a landing page that directs developers to all the Swagger front-ends for all 2017 APIs.

IIRC, this was the still-under-construction http://service.civicpdx.org site, and at one point I believe there was intended to be a static web page at http://service.civicpdx.org/endpoints which would link directly to each of the service front pages e.g. mention the 2017 Portland Budget APIs and link to http://service.civicpdx.org/budget/ (which is still alive and responding).

I don't have a strong opinion about where or how this gets published, but a couple of ideas to get things rolling:

1. static web page routed off the core civicplatform.org such as civicplatform.org/APIs/index.html
2. separate subdomain with a static index.html such as api.civicplatform.org/index.html
3. keep the service.civicpdx.org site and hang a static index off that e.g. service.civicpdx.org/index.html
4. keep the service.civicpdx.org site (since it's hard-coded in all our front-end sites now) and just redirect from an index.html to a more explicit domain name that we'll use in the future e.g. host service.civicpdx.org/index.html, and embed a 301 permanent redirect to apis.civicplatform.org/index.html

Note: if we end up redirecting the service.civicpdx.org domain to api.civicplatform.org or anything else, we'll also have to reconfigure the Host config in all the CloudFormation configuration as has been implemented here.

[nam20485]

Is there any reason to require having a document off of the main URL? Why not have the root point to the static index? That way people with the URL are directly presented with the endpoint options when they browse to the host, without having to know or input more URL. Either that or a landing page with branding at the root with a link to the endpoint list.

All this is to say that I vote for options 2, 3, or 4.

On Sat, Jun 23, 2018, 11:43 AM Mike Lonergan notifications@github.com wrote:

In 2017 season there was an effort (incomplete) to publish a landing page that directs developers to all the Swagger front-ends for all 2017 APIs.

IIRC, this was the still-under-construction http://service.civicpdx.org site, and at one point I believe there was intended to be a static web page at http://service.civicpdx.org/endpoints which would link directly to each of the service front pages e.g. mention the 2017 Portland Budget APIs and link to http://service.civicpdx.org/budget/ (which is still alive and responding).

I don't have a strong opinion about where or how this gets published, but a couple of ideas to get things rolling:

• static web page routed off the core civicplatform.org such as civicplatform.org/APIs/index.html
• separate subdomain with a static index.html such as apis.civicplatform.org/index.html
• keep the service.civicpdx.org site and hang a static index off that e.g. service.civicpdx.org/index.html
• keep the service.civicpdx.org site (since it's hard-coded in all our front-end sites now) and just redirect from an index.html to a more explicit domain name that we'll use in the future e.g. host service.civicpdx.org/index.html, and embed a 301 permanent redirect to apis.civicplatform.org/index.html

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/hackoregon/civic-devops/issues/185, or mute the thread https://github.com/notifications/unsubscribe-auth/ABvqFI9YvhoeLMCVWV45w7EuiBqxXlP-ks5t_ow2gaJpZM4U05FK .

[MikeTheCanuck]

Further digging uncovered the prototype "API directory page" here: https://github.com/hackoregon/endpoint-service-catalog/blob/master/index.html

Which I've connected to the related resources that are/were? responsible for publishing this: https://github.com/hackoregon/civic-devops/issues/183#issuecomment-399701731

However, it appears this is in a state of disrepair - the frontpage for service.civicpdx.org is still under construction, and there's no way for users to easily find the http://service.civicpdx.org/endpoints page (unless they happen to stumble across this very issue).

There's probably a more effective way of advertising the presence, intended usage and backing source code for each of our APIs than this legacy dog's breakfast.

[znmeb]

I'm curious to know how this is going to play out for consumers of the API. I was hoping that I'd get some free time this summer to write some analysis demos against the Transportation Systems API, in particular the congestion data (trimet_stop_events).

[MikeTheCanuck]

service.civicpdx.org/ container

From everything I have deciphered, the container (deployed in ECS as civic-lab-service), to which the requests for http://service.civicpdx.org/ are routed, has no CI/CD origin - there's no repo I can find that creates this container, connects through Travis for building or deploying the container.

Instead, I'm going to guess that Dan Carr built the container by hand from this civic lab service Dockerfile, and then deployed it by hand to the ECS service. Which is obviously an unsupportable path, but I might have the energy to perform this manual task again while we get a better solution deployed via the civic monorepo.

[MikeTheCanuck]

/Endpoints/ container

In the meantime, I think we can get the Endpoints container deployed to update the /endpoints/index.html page content:

• update the endpoint-service-catalog repo with the content I added to PR 47
• create an endpoint-service-catalog repo in HackO's AWS ECR
• update the Travis settings to deploy this container to the new ECR repo (rather than dancarr's personal repo)
• update the push.sh script to current requirements and patterns
• push a PR to the github repo, then merge it so we get the updated container deployed

[MikeTheCanuck]

Refactor the two containers into one

OR - crazy thought - why not just re-imagine the endpoint-service-catalog repo as the home for a single container that hosts the endpoint catalog page, and serves it up not at the /endpoints/ route but rather to the default / route at http://service.civicpdx.org?

What would this actually take?

• [x] revisit the endpoint-service-catalog index.html page to ensure the content fits with its new URL
• [x] create a new ECR repo called "production/endpoint-service-catalog"
• [x] update Dockerfile to eliminate reference to the /endpoints folder (in case there's some dependency between the internal structure of the container and the external routing)
• [x] update Travis project's Settings to publish this container to the new ECR repo
• [ ] deploy the container via Travis to ECR
• [ ] update CloudFormation templates - configure master.yaml to point Endpoints service to / path and remove the CivicLabService, then configure endpoint-service/service.yaml to update the Path, the TaskDefinition > Properties > ContainerDefinitions > Image and the TargetGroup > Properties > HealthCheckPath (as well as add a Description)
• [ ] execute a ChangeSet in CloudFormation to download and initialize the container to its reconfigured ECS service
• [ ] delete the previous ECR repo "integration/endpoint-service"

[nam20485]

+1. It would be pretty nice to have a catalog of endpoints/services right at the root. Then we could put whatever page we wanted there. Endpoint list, or branding + description + endpoints if we ever get that. Whatever, point is we could put whatever static landing page we wanted there and change it at any time.

On Sun, Jul 1, 2018, 9:01 PM Mike Lonergan notifications@github.com wrote:

OR - crazy thought - why not just re-imagine the endpoint-service-catalog repo as the home for a single container that hosts the endpoint catalog page, and serves it up not at the /endpoints/ route but rather to the default / route at http://service.civicpdx.org?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/hackoregon/civic-devops/issues/185#issuecomment-401666310, or mute the thread https://github.com/notifications/unsubscribe-auth/ABvqFACidclkJfN2pcSO8ZuhgOweGmyfks5uCZscgaJpZM4U05FK .

[MikeTheCanuck]

Well I'm stumped. I've:

• updated the index.html file for the endpoint-service-catalog repo
• fixed the Travis build & deploy so that new task definitions are flowing through
• observed the new Task definitions succesfully being scheduled in by ECS

And yet the content of http://service.civicpdx.org/endpoints/ is still on the old page content (only listing the 2017 APIs).

Perhaps the fact that I've pulled out the /endpoints configuration from Dockerfile (at least, as compared to the container that was manually deployed last year) has borked it so that there's nothing to answer to the /endpoints/ route any longer, and so somehow the ALB load balancer is still caching the response that it had from five containers ago?

[MikeTheCanuck]

Result: nope, adding that folder back to the container didn't seem to change a thing:

10.180.8.252 - - [04/Jul/2018:22:47:41 +0000] "GET /endpoints/index.html HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36" "97.115.77.39"

[MikeTheCanuck]

Think I finally figured out what I was missing:

For all the work I did getting a new container built and deployed, I was never able to get the new container's content to return for requests after every CloudFormation update - it was always returning the old content.

I thought I'd updated everything but of course the most obvious thing I completely overlooked: I'd taken the time to create a new, more descriptively-aligned ECR repository naming scheme for the updated container, gotten that to publish to ECR using Travis...and overlooked the now-obvious dependency of actually telling CloudFormation to use the new container image.

How embarrassing.

Check out PR 49 - amazing how easy the solution is once you find it.

[MikeTheCanuck]

OK so that got our new API directory published at http://service.civicpdx.org/endpoints. Proves I now understand how to orchestrate between Git, Travis, ECR and CloudFormation.

Next is to migrate the service so that it's listening at http://service.civicpdx.org/ and replaces (obviates) the Civic Lab Service (which I hope to retire, per #183).

Migration takes two parts:

• change the ListenerRule and related bits (plus removing the civic-lab-service files since we don't need them anymore), per PR 50
• update the container to place the index.html file at the root location of the web server, per PR 10.

[znmeb]

2017 transportation is erroring - the others look OK

[MikeTheCanuck]

And with that, we have a directory of all 2018 and 2017 Django APIs available at

http://service.civicpdx.org/

Is it worth it to try migrating this to http://api.civicplatform.org/ ? Really unclear - still don't have clear idea of the intended evolution and relationships between civicpdx.org, civicplatform.org, the Portland content on civicplatform (vs other cities in the future) and whether we ultimately care more about the APIs that have been delivered, or the platform (monorepo, API exemplar and infrastructure) that can be abstracted from all this.

Thus I'm going to consider this issue "solved" for the moment and fire up new work only when there's a clear direction to follow.