Clean up app+extensions setup and dockerize the stack

[bdewilde]

changes

very sorry to dump a bunch of changes together like this 😞 i was obliged to iteratively reorganize and update various aspects of the app's setup+structure in order to make progress on dockerizing and testing the app, which remains my primary goal.

• main flask app now initialized in app.py, various extensions initialized in extensions.py, and errors initialized separately on a Blueprint, to greatly simplify and standardize the "app factory" code
  • note that i've left the original __init__.py code as a temporary reference, just commented out
  • also note that i've moved the get_uploaded_fulltext_file() route from that file into the api where other similar routes are defined
• drops flask-script dependency and its manage.py interface, in favor of flask's more modern, built-in click cli functionality via cli.py; behavior is effectively unchanged, but the calls are more intuitive
• updates celery app and worker initialization in a way that canonically works with flask app factory style; basically, it's no longer totally hacky
• refactors API setup and structure, to avoid circular imports and more fully embrace app factory style
  • main api object now initialized at the top of the colandr.api package
  • all resources -- including errors and docs models -- now organized using flask_restx.Namespaces (the api equivalent of flask.Blueprints)
• adds more test fixtures and a couple unit tests; user authentication remains a nasty sticking point blocking a full test suite, for now
• updates docker config, particularly the compose services, so that they actually fully work
  • bonus: adds a healthcheck for the db service

questions

• i'm not sure what to do about user authentication, especially in the context of unit testing. any idea how to make http basic auth work from within a routes test? i've burned enough time failing to accomplish this that i'm considering just cutting over to flask-login or something ... would love another perpsective on this problem.
• i'm not sure how docker is handling the postgres database. like, is it persistent across runs of the containers? if so... where are the raw psql data files?? how can i spin up a temporary psql db just for unit testing? what's the best way to seed a fresh db at the start of testing, so i have something to execute client requests against?
• ~i'm also a bit unsure about how docker is networking(?) the various services. are they actually talking to each other, and doing what i want them to do? for example, i'm pretty sure my redis setup is broken, just a bit hard to understand what's going wrong.~ i figured this out! turns out the problem is in the assignment of REDIS_CONN in tasks.py: that redis connection uses localhost by default, and even when specifying a host name that works in the compose file, it doesn't appear to work here. The only way I got this to work was by removing the lock entirely, which is only a temporary fix.

btw, this is where i landed on what was originally posted here: https://github.com/datakind/permanent-colandr-back/pull/48

[bdewilde]

@samanz i think i got the everything* working in docker! to test for yourself, you'll have to comment out all of the lock functionality in tasks.py (ugh sorry), then get the services up and running with docker compose up --build. shell into the colandr-api container and run flask reset-db followed by flask add-admin --name [NAME] --email [EMAIL], plus an immediate password prompt. then you can try running my old "test" script outside docker like python repopulate_db --email [EMAIL] --password [PASSWORD]. you should see lots of logging in the api as well as worker containers that confirms things are chugging along.

*excluding ~full-text uploads~, redis task locking, and user auth in unit testing, as mentioned in the PR message; ~as well as citation deduplication, which should be easy to fix (just have to get the model files into the docker container)~

imho this is a big improvement over the old setup!! just gotta get a couple key missing pieces working before we can really start hacking on this code.

[bdewilde]

update: full-text uploads and citation deduping now appear to be working 👍

[bdewilde]

hi @samanz , i think i've gotten everything working, except for the unit testing user auth issue. I'm just going to merge, so I can work on the testing issue separately. Please feel free to review this PR after the fact.