unity-sds / unity-cs

Unity Common Services
Apache License 2.0
0 stars 2 forks source link

HTTPD configuration Management Improvements #403

Open galenatjpl opened 6 months ago

galenatjpl commented 6 months ago

NOTE: @jpl-btlunsfo before starting on this, we should discuss more..

Investigate how VENUE HTTPD configurations are managed, and improve this process.

Look into whether EFS is still necessary. Mike has recently said that he successfully was able to add a path to the HTTPD configuration via the lambda. As this seems to work, Brad should probably continue along the lines of this approach, and reach out to Mike if he needs an example of how this was done. See writeup: https://github.com/unity-sds/unity-proxy/. See also: https://jpl.slack.com/archives/C028NP76HPA/p1716321080117919

Configurations can be in Github? SSM-change can perhaps trigger a ECS restart? Fetching on start from SSM might be a better approach.

jpl-btlunsfo commented 6 months ago

@galenatjpl per slack, you wanted to discuss this more?

Looking at the current implementation, I feel like the not-really idempotent application of changes is something that is being overlooked. That methodology is what causes the heartburn of testing changes and updating the source config, because the current setup can't purge things from the config- since it just mindlessly injects strings into a file to begin with. The proper way would (in my opinon) involve the proxy refetching the whole configuration from a single source of truth that is actually composable and modifiable.

That the config is stored on EFS doesn't really even buy us anything, and again causes more issues when we're trying to develop changes.

The process I would expect (and hope to implement) would be: 1) ECS httpd container starts up a) httpd container (modified) entrypoint script queries a specific SSM parameter path (based on venue, etc) b) the entrypoint script takes those input snippetss, and based on simple ordinal naming (say 01-ui-proxy, 04-sps-proxy, etc for each) templates its configuration file as part of the startup

2) some application adds a new config snippet to that SSM path area a) lambda catches the change, and triggers a rolling restart of any/all containers (i'm not sure if there are multiple instances of this httpd ECS container) b) container restarts, fetches and templates config file again

1a hinges on being able to query all parameters under a certain path; if we can't do this then i'd say we should keep a list in one parameter that references the paths of the other parameters, etc

2a hinges on the lambda being able to "watch" a SSM parameter change. If we can't do that, then i'd say we should stash the config snipet in some S3 bucket or the like, where it can.

as it is currently, the process seems to be 1) make the lambda create the config file on EFS a) and if it exists just shove the lambda input on top of it b) restart the container?

mike-gangl commented 6 months ago

I agree this is needed at some point, but i think this is an issue in dev for the most part, not in ops. No one, besides me, has even tried using this lambda management function. I agree it's not ideal, but its also really only going to be mis-used or abused in dev. once we get to test and prod venues, these things should be worked out and the chance of adding an "unknown" string into the lambda seems small.

All that is to say, while we want to make this change, i don't think it's that critical to make it now. I'd rather focus on integrating the existing updates into terraform code than making yet another change for something no one is actually using right now.

jpl-btlunsfo commented 6 months ago

Well, I hyperfocused a bit too much on this late yesterday so I've already got some work put in it. If we're tabling it for the future then I guess i'll hold off on a PR.

https://github.com/unity-sds/unity-proxy/tree/403-ssm-httpd-config

galenatjpl commented 6 months ago

@jpl-btlunsfo @mike-gangl let's just discuss more today at 1:30

galenatjpl commented 6 months ago

Hi @mike-gangl : @jpl-btlunsfo and I met yesterday, and decided that @jpl-btlunsfo is proceeding with the plan he laid out (e.g. the PR above). ETA for completion is next week.

jpl-btlunsfo commented 6 months ago

@mike-gangl Would it be possible for you to trigger a docker image build for some new tag (test, 0.13-test, whatever) using the 403-ssm-httpd-config branch, so that I can designate that during testing?

I'm unable to trigger that workflow myself, so it's kinda blocking testing my changes.