AllenInstitute / datacube

Other
0 stars 1 forks source link

move conn bridge into web resource and improve logging #98

Closed chrisbarber closed 6 years ago

chrisbarber commented 6 years ago

With using the reverse proxy, couldn't figure out a way to get crossbar not to log start/stop proxy lines on every request (including health checks). They are only emitted at the "info" log level so we could just bump up to a higher log level in production. But since it's nice to keep the "info" level informative rather than full of noise I pursued this: running the bridge inside crossbar as a twisted resource prevents those extra log lines produced by the reverse proxy resource. Also added a Klein error handler to format failures to the log and include the request that caused it.

conn_bridge still works from the command line as well, as far as I was able to test.

This PR is not ready though because nowhere is the pipenv for conn_bridge being activated. I'm not sure if there is a simple solution to this. Right now things are set up so that datacubes and locator all share a conda environment, which they rely on by crossbar being launched within it. Conn bridge makes an exception by spawning within its own pipenv. The better set up might be (say using pipenv across everything): one pipenv for crossbar+resources (conn bridge would be a resource in this case), additional pipenv(s) for services that are run as guest workers. @NileGraddis - can you think of anything more clever than this / does this sound reasonable?

Will resolve #84.

NileGraddis commented 6 years ago

@chrisbarber your solution sounds fine to me. The conn_bridge shares its dependencies with datacube pretty heavily anyways. Separating these would be nicer of course, but we are at the mercy of the options that crossbar configuration provides.

NileGraddis commented 6 years ago

Also - the changes look good so far. Let me know when you want me to review the full PR.

chrisbarber commented 6 years ago

@NileGraddis I think this is ready to go.

Thought about pipenv across the board but I think conda has enough functionality. I'd like to see conda supporting --require-hashes and an analog to pipenv check. But hesitant to drop conda when depending on things like numpy and scipy. And there are people trying to use conda and pipenv in parallel but I'd rather just use one thing until there is a clear direction on whether they are meant to interoperate.

I like the idea of isolated envs for each service/component but it's simpler to just stick with a single pinned production environment until there is a clear need to break them up for deployments. Top-level requirements are tracked per-service so it should be possible to break them up in the future.

Another thing I'm sad to not be able to use is the annotations that pip-compile (from pip-tools) produces. It would make it easier to excise individual top-level requirements out of a pinned set of dependencies, if that was ever a necessity. I doubt conda will ever have this and it'd be a little messy to try to use them together. Pipenv uses pip-tools under the hood but I'm not sure if it can output annotated requirements.