sites: Have option to not normalize satellite data

openclimatefix / ocf_datapipes

OCF's DataPipe based dataloader for training and inference

MIT License

13 stars 11 forks source link

sites: Have option to not normalize satellite data #361

Closed peterdudfield closed 2 months ago

peterdudfield commented 2 months ago

Detailed Description

It would be great to have an otion to not normalize the satellite data

This is currently dont here - https://github.com/openclimatefix/ocf_datapipes/blob/4bf534e999c6b28efb7889587adbfddc1d007cb0/ocf_datapipes/training/pvnet_site.py#L276

Context

some satellite data is already between 0 and 1, and we dont need to

Possible Implementation

have an option in the config

peterdudfield commented 2 months ago

This will be critical for https://github.com/openclimatefix/india-forecast-app/issues/78

peterdudfield commented 2 months ago

or normalize by the differnt values here https://github.com/openclimatefix/dagster-dags/blob/06c52207cd05fb6f7d06c7a5508fecbe6dbeeb69/containers/sat/download_process_sat.py#L135

peterdudfield commented 2 months ago

I think the idea is to

Have option in config how to normalize, RAW, Normalize, None - Normalize is the current way
include this option in the site datapipeline,
include in other pipelines

Currently in India pipline

%% Flowchart
graph TD
A[Training dataset] --> T1[Scale by min and max, process script] 
T1 --> T2[No, scale by mean and std, ocf_datapipes]
T2 --> B[Batch]
L[Live Satellite Consumer,] --> T3[Add: Scale in ocf_datapipes by min and max]
T3 --> B[Batch]