ECMWF error

[oceancolorcoder]

dev branch

GetAnc_ecmwf.EAC4_download_ensemble returns error "EAC4 atmospheric data could not be retrieved. Check inputs."

Currently called in L1BQC, but soon to be moved to L1B.

[AlexisDeru]

Bug was just an issue in the path of the API config file.

I add some issue to re-synchronize the fork with Dev, this is why 2 commits a visible here, but the first one is some kind of ghost was discarded. Only the latest one is in the PR.

[oceancolorcoder]

I am still getting the error, "EAC4 atmospheric data could not be retrieved. Check inputs" from the cdsapi.Client

[oceancolorcoder]

c = cdsapi.Client(timeout=5) Traceback (most recent call last): File "", line 1, in File "/Users/daurin/anaconda3/envs/hypercp/lib/python3.10/site-packages/cdsapi/api.py", line 304, in init raise Exception("Missing/incomplete configuration file: %s" % (dotrc)) Exception: Missing/incomplete configuration file: /Users/daurin/.cdsapirc

[oceancolorcoder]

It is looking for the config in my home dir rather than HyperInSPACE...

[AlexisDeru]

In GetAnc_ecmwf.py line 56, I defined MainDir as MainDir = os.getcwd() This should be the HyperInSPACE one, no ? Or do you launch the Main.py from another directory ?

Or is it an difference between Linux and Mac ? "os" method should not be sensitive, right ?

[oceancolorcoder]

That part appears fine: MainDir '/Users/daurin/GitRepos/HyperInSPACE' ecmwf_api_config '/Users/daurin/GitRepos/HyperInSPACE/.ecmwf_api_config' os.path.exists(ecmwf_api_config) True

Problem arises at L136 c = cdsapi.Client(timeout=5) That throws the error above

[AlexisDeru]

ok I have reproduced the error, I am looking in detail.

[AlexisDeru]

I do not feel confident enough to modify the cdapi functions. The easy fix would be to copy the .cdsapirc created above into the home directory. Would it be an acceptable solution ? homedir = os.path.expanduser( '~' ) shutil.copy(os.path.join(os.getcwd(),'.cdsapirc'), homedir)

[oceancolorcoder]

I don't see why not, but we need to test on all three OS. I have access here if you want to send update as PR, I can test from here.

[AlexisDeru]

I submit the PR, It runs on my linux VM (unbuntu 20). I check by manually deleting the /home/aderu/.cdsapirc file this time

[oceancolorcoder]

Great. Now comes a sticky wicket. I've been simultaneously making changes that I have not committed or pushed, so you won't have them in your PR. I am going to commit and push and see if the PR will still merge without destruction...

[AlexisDeru]

ok if needed, you can delete my version, it is literally 3 lines of code. So not a big deal

[oceancolorcoder]

Seems to have worked using a merge commit instead of rebasing the PR. Excellent. I will test the model when I get a chance on Win/macOS/linux and mark resolved if successful.

[oceancolorcoder]

First test fails on macOS for file FRM4SOC2_FICE22_UT_20220719_080000_L1AQC.hdf: ECMWF data for Wind and AOD may be used to replace blank values. Reading in model data... 2023-06-06 09:55:19,565 INFO Welcome to the CDS 2023-06-06 09:55:19,565 INFO Sending request to https://ads.atmosphere.copernicus.eu/api/v2/resources/cams-global-atmospheric-composition-forecasts 2023-06-06 09:55:19,740 INFO Request is queued 2023-06-06 09:55:20,921 INFO Request is failed 2023-06-06 09:55:20,922 ERROR Message: the request you have submitted is not valid 2023-06-06 09:55:20,922 ERROR Reason: There is no data matching your request. Check that you have specified the correct fields and values.

[AlexisDeru]

ok perfect, For information, this way of doing thing seems better on my side as it "validate" my commit, otherwise they are still considered "in advance" in my dev branch, even after you accepted the PR, and will be there on the next PR. (Otherwise i have to restart the fork from the begining)

[AlexisDeru]

Ah, ok this is another issue ...

[AlexisDeru]

I suspect an issue in the "pathOut" to write the file in Data/Anc. If you add this lines in GetAnc_ecmwf.py line 141, what do you obtain ? print(pathOut) print(list(EAC4_variables.keys()))

On my side I got the following /mount/internal/work-st/projects/tartu-1281/FRM4SOCv2/workspace/aderu/nasa/HyperInSPACE/Data/Anc/EAC4_E0124N452_20220719120000.nc ['10m_u_component_of_wind', '10m_v_component_of_wind', 'total_aerosol_optical_depth_550nm']

[oceancolorcoder]

print(pathOut) /Users/daurin/GitRepos/HyperInSPACE/Data/Anc/EAC4_E0124N452_20220719060000.nc print(list(EAC4_variables.keys())) ['10m_u_component_of_wind', '10m_v_component_of_wind', 'total_aerosol_optical_depth_550nm']

[AlexisDeru]

This is strange, it does not try to get the same file, I have the noon file on my side, but you try to get the 6am file EAC4_E0124N452_20220719120000 EAC4_E0124N452_20220719060000

[oceancolorcoder]

That is strange. We're both running it on FRM4SOC2_FICE22_UT_20220719_080000_L1AQC.hdf, right?

[oceancolorcoder]

What's the temporal resolution of ECMWF models?

[AlexisDeru]

I do not remember, I am checking with Gabriele.

If you add this in the print list, what do you have ? I want to check if we input the same thing in the query, or if the divergence comes from before that : print((year, month, day, hour)) print([lat, lon, lat, lon])

I got this, for both trios files actually ('2022', '07', '19', '12') [45.2, 12.4, 45.2, 12.4]

[oceancolorcoder]

('2022', '07', '19', '06') [45.2, 12.4, 45.2, 12.4]

print(timeStamp) 2022-07-19T06:00:00

Backing up to L200: print(timeStamp) 2022-07-19T08:00:10+00:00 print(timeStampEff) 2022-07-19T06:00:00

[AlexisDeru]

ok so there a difference in the computed timestampEff, In ECMWF there 2 file per days, one at 00 and one at 12, The code is supposedly choosing between these 2 options. why do you get a 6, is still a mystery.

print(timeStampEff) 2022-07-19T12:00:00

Could it be a different behaviour of datetime between OS ?

[oceancolorcoder]

Not sure, but there is a problem at L100: print((year, month, day, hour,minute,second)) ('2022', '07', '19', '08', '08', '08') print(timeStamp) 2022-07-19T08:00:10+00:00

[AlexisDeru]

Line 108, if you check timestampEff computation you should have the rounding to 12 hours

print(year, month, day, hour, minute, second) 2022 07 19 08 08 08 print(timeSec) 1658210888.0 print(timeEffSec) 1658224800.0 print(timeStampEff) 2022-07-19T12:00:00

[oceancolorcoder]

print(timeSec) 1658232488.0 print(timeEffSec) 1658224800.0 print(timeStampEff) 2022-07-19T06:00:00

[oceancolorcoder]

Which is different, even though our (incorrect) year,month, etc. agree.

print(timeStamp) 2022-07-19T08:00:10+00:00 Should result in: print(year, month, day, hour,minute,second) 2022 07 19 08 00 10 not: print(year, month, day, hour,minute,second) 2022 07 19 08 08 08

[AlexisDeru]

yeah I see 2 thing. first an error in the timestamp conversion, which come from timeStamp2yrMnthDayHrMinSec

Secondly, a difference in the conversion in timeSec, Do the conversion use Julian reference depending on the OS system ? I know there is a 12 hour shift between European and USA Julian reference, could it be that ?

I will correct first error and check what I get

[oceancolorcoder]

Ok. Please try to sync dev if you can before submitting PR; I've pushed commits.

[oceancolorcoder]

Merged the PR. Same error trying to pull the model.

L113: ECMWF data for Wind and AOD may be used to replace blank values. Reading in model data... 2022-07-19T08:00:10:00:00 2022 07 19 08 00 10 1658232010.0 1658224800.0 2022-07-19T06:00:00

[oceancolorcoder]

Problem appears to have been with the lack of timezone @L100. Marking as resolved.

[oceancolorcoder]

Now getting the same old error in Win11:

ECMWF data for Wind and AOD may be used to replace blank values. Reading in model data... 2022-07-19T08:00:10:00:00 2022 07 19 08 00 10 1658217610.0 1658224800.0 2022-07-19T06:00:00 ... 2023-06-06 15:42:11,471 ERROR cdsinf.exceptions.BadRequestException: There is no data matching your request. Check that you have specified the correct fields and values.

[oceancolorcoder]

And in Ubuntu 20.04.6 LTS:

2023-06-06 19:55:41,170 ERROR cdsinf.exceptions.BadRequestException: Value '10:00' not valid for parameter 'time', valid values are: 00:00, 06:00, 12:00, 18:00 ERROR:cdsapi: cdsinf.exceptions.BadRequestException: Value '10:00' not valid for parameter 'time', valid values are: 00:00, 06:00, 12:00, 18:00 EAC4 atmospheric data could not be retrieved. Check inputs.

[oceancolorcoder]

Confirmed that the only reason it was working on macOS is because I already had the netcdf locally. Once removed, error returns. The way I'd gotten the file was to change hour to = '12' from ='06' before running c.retrieve L146

[oceancolorcoder]

Resolved with commit 8aa2e43d9ba5e33e453757bc254b6647816d21ee Tested on Win11, macOS 12.6, Ubuntu 20.04

Win11 now has an issue with Py6S...