Open khasinski opened 4 weeks ago
This should include:
It should be idempotent (ie. it should allow to refresh the data when you run it once)
Use a JSON column to store the cleaned up data.
Save timestamps (created/updated at), save source of the data (PR link, repo name).
This is the first model, so the setup should be SQLite for the database.
The decision if we want to crawl the page or use the API is left to the person implementing this feature.
https://gist.github.com/Vadser/6cae9e9bf3d22def1f128ae18ccbdc17
anatoliliotych
commented
3 weeks ago anatoliliotych
commented
3 weeks ago
This should include:
It should be idempotent (ie. it should allow to refresh the data when you run it once)
Use a JSON column to store the cleaned up data.
Save timestamps (created/updated at), save source of the data (PR link, repo name).
This is the first model, so the setup should be SQLite for the database.
The decision if we want to crawl the page or use the API is left to the person implementing this feature.