search

CNES / MAJA

Level-2A processor used for atmospheric correction and cloud-detection. The active repository is the one below, this one is kept to leave access to the older issues.
https://gitlab.orfeo-toolbox.org/maja/maja
Apache License 2.0
137 stars 25 forks source link

CAMS near-real-time data for the current day (last 24h) #65

Open jprankl opened 4 years ago

jprankl commented 4 years ago

Dear Oliver,

I am currently integrating CAMS data into our MAJA processing pipeline. In principle, I am using the provided scripts to incrementally download CAMS data from ECMWF (MARS) and process new tiles. I found out that CAMS data for the past 24h is not available, but it seems (at least some) data is accessible from an ECMWF FTP server.

So, my question is if it is possible to immediately process Sentinel tiles when they are available (about 6h after sensing) using CAMS?

Is the needed CAMS (surface, model, pressure) data available on the FTP server? Can that data be used?

Best regards!
Johann

jprankl commented 3 years ago

Dear @olivierhagolle and @petket-5,

it seems there is a new API where cams data can be accessed (https://ads.atmosphere.copernicus.eu) and at least the ECMWF public web API is stopped on June 30th, 2021.

Is it possible/ recommended to use the new API? How would the request look like?

Thanks!

olivierhagolle commented 3 years ago

Yes, the change comes with short notice :( The team is currently working hard to adapt to the new API. Olivier

jprankl commented 3 years ago

Thanks for the fast reply. Indeed the ECMWF mail was short notice and even a little bit confusing to me.

jprankl commented 3 years ago

Dear @olivierhagolle and @petket-5 ,

yesterday I did a quick test with the new cams downloader from Maja 4.2.7. Two questions came up:

I am using Maja in a (touched) real-time style, so the last 7 days are important for me. Do you know about that issue? Is that because of the new API? What would you suggest, do I need to cope without CAMS data?

Thanks in advance! Johann

jerome-colin commented 3 years ago

Dear @jprankl, I did several tests on different tiles and got the same issue : no CAMS found for recent days, while I can get them through the cdf api. There's most probably a bug, I'm opening an issue. Thanks for raising this problem, I'll come back to you asap. Jerome

jerome-colin commented 3 years ago

Hi @jprankl , Unfortunately, the problem doesn't seem to be a bug, but a license issue. While the aerosols AOD and mixing ratios are computed by CAMS, the relative humidity is a meteorological field from ECMWF. And since ECMWF doesn't belong to the general Copernicus agreement, the access to this data is delayed by 5 days. That's a pretty serious obstacle to activating the --cams option for near real time processing. We'll let you know if we find a workaround.

jprankl commented 3 years ago

Hi @jerome-colin ,
many thanks for your investigation!
Is the use of humidity new in 4.2.7? At least in Maja 3 and the old API I did not have that issue.
Would be cool if you find a workaround and let me know. Thanks!

jerome-colin commented 3 years ago

We've been using relative humidity with cams for more than 2 years, but I never tried near real time runs for algorithm validation. I cannot tell whether it comes from the latest API or if we just never came into this issue because of the time period of the contexts we usually processed. ECMWF data are accessible via institutional agreements, but even so it'll need some additional tuning from our side to plug an additional data source. Jerome

jprankl commented 3 years ago

o.k., please let me know if you find some options.
thanks for your effort!