Update and better define (dev) app environment

[bdewilde]

changes

• bumps Python version to v3.7.X
• bumps PostgreSQL version to 10.X
• adds setup files to make colandr into an installable Python package
• adds pyproject.toml with package tool configurations
• bumps / constrains versions of key dependencies
  • celery from 3.1.25 to ~4.0
  • dedupe from >=1.4.14 to ~1.4.14
  • itsdangerous from >=0.24 to >=2.0.0,<2.1.0
  • psycopg2 from >=2.7.0 to ~=2.9.0
  • redis from ==2.10.5 to ~=3.2.0
  • spacy from >=1.2.0,<2.0.0 to ~=2.0
  • textacy from >=0.3.2,<0.5.0 to ==0.5.0
  • webargs from >=1.5.1 to ~=1.5
• makes minimal changes to textacy usage in code to agree with version bump
• changes localhost port, from 5000 to 5001, since recent versions of macOS claim 5000 for themselves
• updates .gitignore to include standard python packaging cruft
• updates dev env setup docs to follow more modern conventions/requirements

context

afaict, on the production machine, we are currently running Python 3.6.12 and PostgreSQL 9.6.11 with the following dependencies (filtered to just direct, not transitive, deps)

alembic==1.0.9
arrow==0.13.1
bcrypt==3.1.6
bibtexparser==1.1.0
celery==3.1.25
dedupe==1.9.6
Flask==0.12.4
Flask-HTTPAuth==3.2.4
Flask-Mail==0.9.1
Flask-Migrate==2.4.0
Flask-RESTful==0.3.7
flask-restplus==0.12.1
Flask-Script==2.0.6
Flask-SQLAlchemy==2.3.2
gunicorn==19.6.0
itsdangerous==1.1.0
Jinja2==2.10.1
MarkupSafe==1.1.1
marshmallow==2.19.2
numpy==1.16.2
psycopg2==2.8.2
python-dateutil==2.8.0
python-redis-lock==3.1.0
PyYAML==5.1
redis==2.10.5
requests==2.21.0
scikit-learn==0.20.3
SQLAlchemy==1.2.19
textacy==0.4.2
webargs==5.3.0

i wasn't able to reproduce this environment locally, owing to changes in supported versions of things (e.g. PY3.6 no longer supported properly via homebrew installs), so i had to incrementally bump / constrain things until everything got working.

questions

• how do i get pdfestrian running locally? i followed the instructions in its readme, but was not able to get it going. predictably, calls to a certain colandr endpoint failed: [2023-04-17 21:26:44,208] ERROR in errors: an unhandled exception occurred: Command '['/Users/burtondewilde/Desktop/projects/datakind__colandr/permanent-colandr-back/pdfestrian/bin/extractText.sh', '--filename', '/Users/burtondewilde/Desktop/projects/datakind__colandr/permanent-colandr-back/colandr_data/fulltexts/1/44.pdf']' returned non-zero exit status 1.
• how do we safely update the production environment, ideally without any disruption in app service? (i realize that "no disruption" may not be possible, considering the significance of the env changes.) like... what is the "deployment" process here?
• not sure if this is a crazy idea, but... what if we just created an entirely new version — like v2.0 — of the colandr back-end, using all modern tools, and didn't worry about compatibility / upgrade paths for the current v1.0? at some point we'd basically snapshot the prod database and load it into a fresh one, then spin up the new back-end in the old one's place? this is crazy, right?

[samanz]

I'll work on a PR for updating pdfestrian, and make sure it all works.

[bdewilde]

@samanz not a big deal because i've not added branch protections yet, but in future PRs, please don't hesitate to approve/request changes rather than just comment. (unless only commenting is your intention, of course.) hopefully that will help speed up review cycles, especially since our dev hours are bound to be on different schedules for this project. sound good?