jpwahle / cs-insights-backend

API server of the cs-insights project. This is the main part of storing data and accessing an external data analysis endpoint. It uses a mongoDB instance to store everything and queries the cs-insights-prediction-endpoint to get machine learning results.
https://jpwahle.github.io/cs-insights-backend/
MIT License
7 stars 0 forks source link

Precompute values #49

Closed trannel closed 1 year ago

trannel commented 2 years ago

Is your feature request related to a problem? Please describe. The stats endpoint needs to know certain values, which are much easier to precompute than to compute each query. Currently those values are hardcoded.

Describe the solution you'd like Find an automated solution, if possible: MongoDB should precompute those values each time it starts running. The backend can access the values and use them in the correct locations. Or let the crawler write the values to a file each time the crawler runs. The backend can access this file and use the values in the correct locations.

Describe alternatives you've considered If no automated solution can be found, notify the admin about the changes that need to happen.

jpwahle commented 2 years ago

I marked this as duplicate just because it won’t be an issue anymore after caching in #89 and fixing database schemas https://github.com/gipplab/cs-insights-backend/issues/90

trannel commented 2 years ago

The issues listed here are things that are currently hardcoded, as the issue says, and need to be changend at some point or there will be strange issues, when new data comes in. The idea was to procompute them. Caching and different database schemas will not fix this.

jpwahle commented 2 years ago

I see, I will reopen it in this case and make sure that instead of your hardcoded time range implementation, we will query the DB (which then will be cached).