peterdudfield
commented
1 year ago The simplest answer is because its simple. Although simple isnt always the best, it does help reduce tech dept and keeps things more readable e.t.c. Simon started with something like this but then started saving night time zeros, in order to keep things simple.
However, we might want to consider this to reduce the total about of data saved.
Some thoughts:
Dont save zeros
- We could change the save method, to not save night time zeros - that wouldnt be too hard. I would be cautious about filling in zeros in the API. This might take more time (to work this out for 318) localisation than just keeping the zeros.
- Edge case: We would have to think about edge cases where the forecast is missing but we still want to fill in?
- Edge case: Also need to think about edge cases like the forecast is only 8 hour long, the night is 12 hour long, do we just fill forward? but what happens if the forecast is missing? just keep filling in forward? Maybe a way to save that would be to remove the nightime.
- Edge case: to think might be zero in scotland even though it isnt night time.
- FE filling would be fine, have to catch edge cases, I slightly think this stuff should happen in API and be cached than in the FE
- Would it be weird if a user was using the API and there were missing values not zeros, I think so. This would mean doing some work in the API.
Schema change
- We could also change the scheme to include start and end datetime, then would could just have one interval for night time, which would be zero. Im not sure how keen I am to change the schema right now.
Dont Load zeros
- Perhaps we could have an option in the API removes any zero values, then the normal API users could still get it. The Front end could try this, and fill in as needed. The data transfer time would be less too.
- Spike: We could try SQL query where value!=0 and see if it speeds the query up.
- Edge case: to think might be zero in scotland even though it isnt night time.
- I probably like this the both, or atleast do the spike for this and reassess
From @devsjc and @braddf
A potentially dumb question for eurostar-based mulling: why do we bother saving forecast values for time periods outside of local sunrise and sunset? Can you ever have a predicted solar generation of greater than zero say 2 hours after sunset and 2 hours before sunrise? Might be able to save a lot of redundant rows in our database not bothering to save those - the frontend could just fill in zeros for missing values in that time window, which would mean we could still determine differences between missing and zero values that are occurring not at night (so we wouldn't fall into the trap that came up early on when I started and asked why we save zeros at all!)
Brad: I am not entirely sure, maybe there is a reason for it, or maybe it’s just to keep all the forecasts standard independent from their creation time… but I agree that would save a whole bunch of rows - although I would say we could probably just “fill in” the night numbers on the API side fairly quickly and easily for each set of forecast values, and then the API users and the UI get exactly what they have now
Sol: Yes, perhaps it makes more sense to do the filling in on the API side