search

carbonplan / offsets-db-api

accounting of voluntary and compliance offset programs
https://carbonplan.org/research/offsets-db
MIT License
1 stars 0 forks source link

alembic migrations causing build failures on fly.io #72

Closed andersy005 closed 9 months ago

andersy005 commented 9 months ago

we are currently encountering an issue where running Alembic migrations (python -m alembic upgrade head) leads to build failures. the process exits with a "double free or corruption" error, which points towards a memory management issue. 

  double free or corruption (out)
  release.sh: line 14:   326 Aborted                 python -m alembic upgrade head
   INFO Main child exited normally with code: 134
   INFO Starting clean up.
   WARN hallpass exited, pid: 306, status: signal: 15 (SIGTERM)
  2023/12/06 01:59:01 listening on [XXXXXX]:22 (DNS: [fdaa::3]:53)
  [    6.276142] reboot: Restarting system
  machine restart policy set to 'no', not restarting
Error: release command failed - aborting deployment. error release_command machine YYYYYYYY exited with non-zero status of 134

this error does not occur when running simple Python commands, indicating the problem is specific to alembic or its interaction with our environment.

as a temporary measure, i have disabled alembic migrations to prevent build failures. this is not a sustainable solution as it bypasses important database schema updates. i'm opening this issue to keep a paper trail and ensure this temporary measure is well documented

andersy005 commented 9 months ago

it turns the culprit was the psycopg2 dependency which was causing memory corruption. installing just the binary https://pypi.org/project/psycopg2-binary/ fixes the issue...