Consolidate OONI API

[FedericoCeratto]

• [x] Merge API repo in subdirectory in the pipeline repo
• [x] Refactor API codebase in a dedicated directory
• [x] Remove mr_table related stuff
• [x] Cleanup dependencies, add new dependencies for Authentication
• [x] Create package
• [x] Deploy on new pg11 host
• [ ] Implement auth #391 (compatible with the PS login/register calls)
• [x] PS: Implement collector create/close report
• [x] PS: Move Prio into a test-list call
• [x] PS: Add bouncer
• [x] Consolidate Probe Services
  • [ ] Authentication
  • [x] Collector
  • [x] Dynamic URL test list with priorities
• [x] Implement new canning-equivalent
• [x] Remove dependencies from batch pipeline
  • [x] Measurement table
  • [x] Reimplement private api to move away from ooexpl_* tables
• [x] Implement packaging to support CI deployment
• [x] Run as a daemon using a systemd unit file
• [x] Cleanup dependencies and use stable libraries from Buster
• [x] Generate API docs/swagger automatically with flasgger
• [ ] Quality:
  • [ ] fix logging
  • [ ] ensure functional tests and coverage are good
  • [ ] chase FIXMEs and TODOs
• [x] implement measurement_meta lookup: autoclaved, minican, fastpath host
  • [x] implement jsonl in uploader
  • [x] update Fastpath to support jsonl cans
  • [x] update Fastpath to support API POST submissions
• [x] fix S3 bucket public permissions
• [x] implement and test cloudfront msmt submission (see https://github.com/ooni/backend/issues/451#issuecomment-701320675)
• [x] implement msmt upload into api with new mini-canning
• [x] implement analysis timers to refresh counters* tables
  • [x] benchmark update/refresh methods
  • [ ] rebuild entire tables once to fix gaps in the last month
• [x] chase Prometheus conf, create API dashboard and alerts
• [x] implement temporary script update autoclaved_lookup every day
• [ ] implement cron timer to update fastpath table from the live fastpath during the migration
• [ ] disable/allowlist throttling for probe-service entry points
• [x] forward from Nginx to orchestra.ooni.nu , test helper & co for entry points: login register psiphon tor-targets
• [x] add DB creation to fastpath
• [ ] test DB creation done by the fastpath on a new host
• [ ] fastpath: add support for fetching new minicans from s3
• [x] end-to-end test all probes
• [x] investigate and implement, test Tor msms submission
• [ ] implement ooni-in-a-box CI
• [x] Add software_name _version to fp and rerun last month
• [ ] Make sure the mobile probes still work
• [x] Stress test of the backend
• [ ] YAML submission: end-to-end test
• [ ] enumerate the places where we MUST keep existing DNS names (e.g. mobile apps)
• [x] Implement fastpath-based API to support explorer (formerly https://github.com/ooni/backend/issues/442)
• [ ] drop unused columns
• [x] move domain filtering to domain column in fastpath
  • [x] populate the database
  • [x] update fastpath to populate the domain column
  • [x] update API to use the domain column
• [x] fix filtering by anomaly in list_measurements
• [x] fix counters table updater
• [ ] set since/until automatically in list_msmts if needed
• [ ] see whether we can deploy a new probe-service in MIA?
• [x] backup strategy for the database
• [x] write migration plan
  • [x] sync autoclavedlookup after migration
  • [x] rerun fastpath after migration
  • [ ] rebuild counter tables
• [x] configure letsencrypt to generate ps1/ps2/api.ooni.io using ansible
• [ ] create ps5test.ooni.io on ams-pg and test certbot
• [x] update timer to fetch citizenlab lists
• [x] flip backend
  • [x] run fastpath manually from S3 cans
  • [x] deploy Explorer (new code + use ams-pg as API)
  • [x] flip ps1.ooni.io
  • [x] run letsencrypt for ps1.ooni.io
  • [x] run miniooni against ps1.ooni.io and check msmt on explorer
  • [x] flip ps2.ooni.io
  • [x] run letsencrypt for ps2.ooni.io
  • [x] run miniooni against ps2.ooni.io and check msmt on explorer
  • [x] flip api.ooni.io
  • [x] run letsencrypt for api.ooni.io
  • [x] test both API and Explorer
  • [x] filp collector.ooni.io
  • [x] run letsencrypt for collector.ooni.io
  • [x] run legacy probe using onion service and check msmt on explorer
  • [x] reconfigure bouncer.ooni.io to point probes to ams-pg
  • [x] run legacy probe using bouncer.ooni.io and check msmt on explorer

PS endpoints: https://docs.google.com/document/d/1WGkKF1dCFQqa-8zFPN9EjiOQldSaotZPDzgClOLZFRw/edit

[FedericoCeratto]

Pull Request https://github.com/ooni/api/pull/192

[FedericoCeratto]

Implemented in https://github.com/ooni/api/pull/192

[FedericoCeratto]

Completed

[bassosimone]

🥳