search

PMA-2020 / pma-api

The best way to programmatically access PMA2020 data.
http://api.pma2020.org
MIT License
1 stars 3 forks source link

Apply staging/production and db backup/restore #57

Closed joeflack4 closed 5 years ago

joeflack4 commented 5 years ago

1. New features

  1. Update to apply staging/production feature (WIP)
  2. Auto DB backup/restore on certain non-idempotent tasks
  3. Database backups can now be stored on S3
  4. API_Data spec datasets now can be stored on S3
  5. UI_Data spec datasets now can be stored on S3
  6. Database restore
  7. Database schema migrations
  8. Modifications to Dataset model
  9. Dev environment management: changed admin page to display 'dev' environment info if env is dev, and made updates to sync status of datasets asynchronously (WIP)
  10. Can now list API_Data and ui_data stored on S3 and locally
  11. RESTful query syntax: Added basic functionality to allow filtering of a resource using Python syntax or Pythonic syntax. Will need to implement some security measure, (e.g. make sure multiple statements cannot be passed using ;, disallow imports, etc). This allows for a very strong query language, specially if certain Python functions are pre-loaded into the query runtime (things like max(), min(),avg())

2. General updates

  1. New API dataset uploaded, tested, and deployed
  2. Dataset integration automation: format Stata '.' to empty value if present in a numeric type field (WIP)
  3. Admin page: Active dataset information now fetched asynchronously from the client, rather than server
  4. Dynamic routing: Basic implementation for dynamic routing to access model resources (e.g. Dataset) that have not yet had a static route created. Planning to do more of this in the future because it is DRYer.
  5. Update to docs

3. Developer updates

1. Dependencies

  1. Added dependency python-dotenv to use '.env' files
  2. Added dependency boto3 for working with AWS S3 for backup storage
  3. Added dependency'psutil for server management; stop/restart server
  4. Migrated deprecated dependency psycopg2 to replacement psycopg2-binary

    2. QA

  5. More test cases
  6. Numerous error messaging additions and improvements

    3. Refactors

  7. Moved all routes into subdirectory 'routes'
  8. Split up root routes into different modules
  9. Moved several things from init to: config, tasks, routes/root_routes, and utils

    4. Other

  10. Removed sqlite alternative DB implementation
  11. Reorganized gitignore, in order of: folders, glob patterns, static files
  12. Linting

1. Bugfixes

  1. Fixed a bug in which, if there were translations entries that were not found anywhere else in the dataset, it would throw an error.
  2. Fixed bug in which some database related error handling was not correctly handling some types of intended errors
  3. Fixed issue in which caching would not work
  4. Fixed issue where Flask environment read as 'production' even though it appeared to be set as 'development'