ecmwf / ecpoint-calibrate

Interactive GUI (developed in Python) for calibration and conditional verification of numerical weather prediction model outputs.
GNU General Public License v3.0
21 stars 8 forks source link

Allow to modify the Units even if the "addition" and "multiply" option are not selected #128

Closed EstiGascon closed 3 years ago

EstiGascon commented 3 years ago

When we load each predictor to create the calibration database, you can only modify the "Units" when you modify the variable using the "multiply" or "addition" option. Due to the fact that we have to pre-compute some variables outside and we cannot modify the units internally in the grib file, we need this option to specify the units in the ascii or parquet calibration file.

FatimaPillosu commented 3 years ago

I just checked whether we can change the units, names, etc in Metview for these pre-processed variables, and we can't. Therefore, we will need to allow the users to change this in the software. I was thinking of an option that can allow the software to recognize in advance the pre-processed variables and then, enable the changing of the units and the names to then to be printed in the PDT.

@onyb , could you let us know whether this would be possible? Let's say we call the directories of pre-processed variable with the suffix "[NameVar_prep]". If there are directories with that suffix, the software opens a box where we can insert the following details:

I think the way Esti is suggesting is one way, but I think it could bit error-prone if we enable this option in general, or at least confusing if in some cases the user can change the units when he/she is not adding or multiplying by a value. This might be a bit more work but I think it will be more consistent with the general approach so far taken. What do you think, @onyb?

I'm changing this into the "Bug Milestone" to add this into the first things to do, but changing to "priority 2" as, although this is high priority anyway, users are still able to use the software.

Cheers,

Fatima

onyb commented 3 years ago

This feature turned out to be a very tricky one, mainly because it involved a lot of logical changes in the way we handle primary predictors (from the filesystem) but also derived predictors (computed from primary predictors). Also, if I understand correctly, we want this feature only to have correct units in the point data table metadata. Please correct me if I'm wrong.

Here's what I managed to achieve.

Step 1: Click on the new "Override predictor metadata" button

Screenshot 2020-12-02 at 02 59 28

Step 2: Modify the metadata as required

Notice how I modified the u700 variable to use a different name and units.

Screenshot 2020-12-02 at 03 02 49

Step 3: Add a computation with the modified variable

The name and units will now use the overridden value instead.

Screenshot 2020-12-02 at 03 05 20

⚠️ Limitation

One limitation we have is that we cannot modify the variable (or, short name) itself. This is mainly because it would involve a massive refactoring, and might break existing functionality. The difficulty arises from the fact that the short name is directly mapped to directories on the filesystem.

Let me know if this is an acceptable solution, and I can push my changes.

FatimaPillosu commented 3 years ago

Hi Anirudha, this is really good! I think this might solve our issue as we can change the (long ) name and the units as needed. The short name we can change it easily because we just call the directory as needed and...Bob's your uncle! But let's see what @EstiGascon and @AugustinVintzileos say. Cheers, Fatima

EstiGascon commented 3 years ago

Hi Anirudha, Many thanks for improving this part! I think that what you applied is really good and it will be enough to fix the units problem. We will test it went he new version is available, but just from your pictures it looks more than enough for me. Thanks! Esti

EstiGascon commented 3 years ago

Hi Anirudha,

I tested this feature, but when I change the names and units, the changes are not applied to the predictors when I leave the Override metadata windows. Is there any "accept changes" button? I just go out clicking outside that windows because I cannot see any button to accept the changes. But the changes are not applied anyway. Thanks! Esti

onyb commented 3 years ago

Hi Esti,

You need to make sure that the units are overridden first, before adding the predictors. If you want to apply the overrides to existing predictors, you'll have to remove and add them back.

Let me know if this is not what you're talking about.

EstiGascon commented 3 years ago

Hi Anirudha,

Many thanks, it worked as you said, so we are changing now everything before creating the predictors.

Cheers, Esti