Cleanup dockerfiles, remove duplicates, upgrade PDM

spwoodcock commented 3 weeks ago

What type of PR is this? (check all applicable)

[ ] 🍕 Feature
[ ] 🐛 Bug Fix
[ ] 📝 Documentation
[ ] 🧑‍💻 Refactor
[ ] ✅ Test
[x] 🤖 Build or CI
[ ] ❓ Other (please specify)

Describe this PR

Removes dup / redundant dockerfiles.
Upgrades PDM and debugpy to latest.
Removes hardcoded --workers flag for Gunicorn (can be set by WEB_CONCURRENCY env var instead).
Re-locks dependencies after PDM version upgrade using inherit_metadata strategy (required PDM > 2.15).

Review Guide

Notes for the reviewer. How to test this change?

Checklist before requesting a review

📖 Read the Tasking Manager Contributing Guide: https://github.com/hotosm/tasking-manager/blob/develop/docs/developers/contributing.md
📖 Read the HOT Code of Conduct: https://docs.hotosm.org/code-of-conduct
👷‍♀️ Create small PRs. In most cases, this will be possible.
✅ Provide tests for your changes.
📝 Use descriptive commit messages.
📗 Update any related documentation and include any relevant screenshots.
🔠 Does this PR introduce or change any environment variables? If so, make sure to specify this change in the description.

[optional] What gif best describes this PR or how it makes you feel?

sonarcloud[bot] commented 3 weeks ago

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

mahesh-naxa commented 2 weeks ago

@dakotabenjamin i just verified this, can be merged.

dakotabenjamin commented 2 weeks ago

Errors after merging, which weren't caught because they are in server instantiation:

+ sudo apt-get -q -y install libgeos-3.9.0 libgeos-dev
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package libgeos-3.9.0
E: Couldn't find any package by glob 'libgeos-3.9.0'
E: Couldn't find any package by regex 'libgeos-3.9.0'
+ sudo apt-get -q -y install libproj19 libproj-dev
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package libproj19

In addition, a slew of errors compiling sklearn during pip install. edit: and a warning that might be related:

/tmp/pip-build-env-ao16q_16/overlay/lib/python3.8/site-packages/scipy/__init__.py:143: UserWarning: A NumPy version >=1.19.5 and <1.27.0 is required for this version of SciPy (detected version 1.17.3)
      warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}"

spwoodcock commented 2 weeks ago

Any way we can test the Cloudformation (I assume) stuff locally?

It's a bit baffling how the libgeos install would have broken when this PR updates dockerfiles 😯

The second issue around the numpy version can be fixed by pinning the numpy version explicitly (if sklearn requires this)

mahesh-naxa commented 2 weeks ago

@spwoodcock We can use https://www.localstack.cloud/ to run the EC2 instance with the defined user-data scripts locally, but i am not sure if it supports the AMI we are using. Let me check. Otherwise i will spin up a EC2 in the AWS to validate everything.

hotosm / tasking-manager