scale fetching process - Githubissues

openpodcast / api

OpenPodcast API server

https://openpodcast.dev

8 stars 2 forks source link

scale fetching process #83

Closed woolfg closed 1 year ago

woolfg commented 1 year ago

add podcasts without deploying its own stack
using one stack/workes for all podcasts

simplest approach:

[x] store all keys in DB
[x] create spotify and apple workers (containers)
[x] the worker fetches all podcasts and its keys and executes the connector in a loop

woolfg commented 1 year ago

simplest approach:

store all keys in DB
create spotify and apple workers (containers)
the worker fetches all podcasts and its keys and executes the connector in a loop

what so you think @mre ?

woolfg commented 1 year ago

main idea implemented in a quick and dirty way openpodcast/pipelines#50

woolfg commented 1 year ago

basic flow is working and in test on the generic infra.

stack looks a way lighter now:

openpod-engineeringkiosk_analytics           replicated   1/1        metabase/metabase:v0.44.6                                         
openpod-engineeringkiosk_api                 replicated   1/1        openpodcast/api:latest                                            
openpod-engineeringkiosk_backup              replicated   1/1        woolfg/mysql-backup-sidecar-gs:0.1.2-mysql-8.0                    
openpod-engineeringkiosk_connector-manager   replicated   1/1        openpodcast/connector-manager:sha-dfa27f1                         
openpod-engineeringkiosk_db                  replicated   1/1        mysql:8.0