Update system diagram for the openFEC system

fecgov / openFEC

The first RESTful API for the Federal Election Commission. We're aiming to make campaign finance more accessible for journalists, academics, developers, and other transparency seekers.

https://api.open.fec.gov/developers

Other

479 stars 106 forks source link

Update system diagram for the openFEC system #1591

Closed NoahKunin closed 8 years ago

NoahKunin commented 8 years ago

There's been a tremendous amount of progress on openFEC, so we need to draw an updated system diagram.

LindsayYoung commented 8 years ago

That makes sense, this project has really taken off. Thanks for making an issue. I have a few follow up questions.

About when is the diagram needed by- 2 weeks ok
Do you want one diagram for all of FEC, each project or one for each repo
What do we do with the diagram once we make it- post it here, send it to you etc.

Thanks

NoahKunin commented 8 years ago

@LindsayYoung

I'd ask it be put in scope whenever your next sprint starts. I wouldn't crack a sprint to add this, there's not that level of urgency. Once it is in a sprint, I would expect it to be done within 2 sprints. So in other words, I won't micro-manage the flow of the work, since there is no artificial external deadline at the moment. :) Just let me know when there's room for it to enter a sprint, and the deadline will emerge from that organically.
Great question. I would like there to be a singular file - but I would make it as big or as many Ya'll are the experts on the system. Think about it this way - let's say everyone currently on the project had to go to Mars tomorrow. If you had to draw a new FEC team a picture of how the system works, would you draw one big diagram or (n) small ones
You can put it where it makes sense to you! However, you just have to ensure the permalink gets into your SSP YAML file here: https://github.com/18F/openFEC/blob/develop/system-security-plan.yml#L53 If a singular diagram effectively represents its functional and networking state, you can use the same link for line 54 - again, depends upon the output of the bullet above.

LindsayYoung commented 8 years ago

Thanks @NoahKunin

LindsayYoung commented 8 years ago

List of things to diagram:

3 RDS instances with assess the internet so the FEC can push updates. Devs need access to Amazon so that they can scale up the instances in times of heavy traffic, like filing deadlines and schema updates. The team also responds to alerts as necessary. These are not hosted on cloud.gov
The API needs S3 access to the fec bucket to create downloadable csv resources. Those resources expire daily.
Celery worker to create and maintain download que and start nightly refresh
Redis swarm is needed to store download que
API needs access to api.data.gov, which is whitelisted from production (Prod only)
Mandrill API access for email error messages from the update processes
Slack access for the apps to post what branch is deployed where
The Proxy app needs access to DNS and to route traffic for beta.fec.gov, which consists of the Web app and CMS
The Web app and CMS need access to Github to post user feedback
The CMS uses a postgres database in cloud.gov
Testing and deploys from Travis on successful builds
Error monitoring: NewRelic, Sentry
FEC legal resources is working on adding an elasticsearch instance for legal documents

noahmanger commented 8 years ago

Thanks for the write-up @LindsayYoung . I'm going to throw this in next sprint and then find time for us to pair on it.

LindsayYoung commented 8 years ago

Sounds perfect

noahmanger commented 8 years ago

Sorry for the delay on this @NoahKunin but here's a first swipe at it with @LindsayYoung .

Here's our production set up at the moment:

fec system diagram

And here's what we have on dev and staging (one for each): fec system diagram - dev

Note the latter includes grayed-out nodes for the upcoming legal resources system. @anthonygarvan and @adborden , anything to add here?

Let me know if this is along the lines of what you're looking for or if you want any clarification or refinements. Thanks for everything!

adborden commented 8 years ago

:+1: Those grey areas are correct.

NoahKunin commented 8 years ago

Great, thanks! What did you use to diagram it out? I might recommend the same to other teams.

noahmanger commented 8 years ago

Omnigraffle. I can share my file if it's helpful.

NoahKunin commented 8 years ago

Indeed, thanks!

LindsayYoung commented 8 years ago

Taking a second look, I noticed some minor things.

The CMS has the same error reporting as the web app: cms to Sentry cms to New Relic

There should be an additional cf credential service. "FEC cred service (CMS)" should be pointing to the cms and the current on should jsut say "FEC cred Service" and that is shared with the API and the web app.

The direction of the arrow between the api and s3 should be reversed, and users access downloads directly from S3 as well.

@noahmanger this looks great!

noahmanger commented 8 years ago

Thanks for the edits. Does this look right @LindsayYoung for production? fec system diagram

LindsayYoung commented 8 years ago

Looks right! Thanks again

porta-antiporta commented 8 years ago

@noahmanger May i get the omnigraffle file? i'd like to update the diagram. This is to satisfy a request from another issue to help us understand what is needed to ship regulations on beta.

NoahKunin commented 8 years ago

fec system diagram.graffle.zip

@porta-antiporta Was on this thread anyway, here you go!

LindsayYoung commented 8 years ago

The most up to date diagram is now: https://github.com/18F/fec-eregs/issues/125#issuecomment-224979847

porta-antiporta commented 8 years ago

attached is the omnigraffle file for that diagram. fec system diagram updated 2.zip

It includes a production view sans legal resources stuff as well.

noahmanger commented 8 years ago

Sounds like everyone has what they need, so closing this.

LindsayYoung commented 7 years ago

fec-system-diagram