Closed meteo1987 closed 2 years ago
guidocioni
commented
2 years ago I'm guessing you modified the name of the environment variable holding the mapbox API key? was that on purpose? Anyway the issue is that it doesn't find that env var, which means that you probably did not define it. Unless you really want to use the mapbox layers you can maybe just define this as empty string like apiKey =""
meteo1987
commented
2 years ago Thank you for your quick response. I try your suggestion and define apiKey="", but the error is still there. Unfortunately my knowledge is not so good, so right now I don't know what to change to get this script work. I attached the cron.log file of last run.
Kind regards,
Sasa
24.01.2022 23:36, je Guido Cioni napisal
I'm guessing you modified the name of the environment variable holding the mapbox API key? was that on purpose? Anyway the issue is that it doesn't find that env var, which means that you probably did not define it. Unless you really want to use the mapbox layers you can maybe just define this as empty string like apiKey =""
-- Reply to this email directly, view it on GitHub [1], or unsubscribe [2]. Triage notifications on the go with GitHub Mobile for iOS [3] or Android [4]. You are receiving this because you authored the thread.Message ID: @.***>
/home/meteolab/icon-d2
guidocioni
commented
2 years ago Sasa,
please take time to understand the errors before posting a full log here: the problems that you're getting are not because of this library but it seems to me to a poor understanding of Python and the environment where you're running the scripts. So it doesn't make sense to jump into a cronjob if you're not sure beforehand that you understand all the parts that are called inside.
As described in the README there are a (1) download/processing (2) plotting and (3) upload step.
Try to investigate all these steps separately before running the whole script again.
It seems that the download for you is working, although I'm not sure what this means
UNCHANGED_RECORD=0But assuming the download and processing are working fine just try to run a single python script, for example python plot_cape.py with the flag debug=True, before running all of them at the same time.
Now focus on the error that you're getting in utils.py, it is coming from this line
If you do this os.environ[' '] you will get an error as there is no environment variable matching that name.
The apiKey value is used to make a dark background in some plots (plot_background_mapbox) and to find coordinate for cities to plot meteograms (get_city_coordinates). As I said, if you don't need these capabilities just comment out the lines using those functions. If you never use plot_meteogram.py you'll never use that key, same if you never use plotting routines that use plot_background_mapbox.
An alternative would be to get an account to mapbox so that you can get an API key.
meteo1987
commented
2 years ago Hello,
I appreciate your time and consideration. I admin that my poor knowledge of Python affected the problems with the operation of the script. Yesterday I manage to successfully generate some images from script. I noticed that some products are "blank" (only borders and legend) or visually unreadable. Is this related to input data or something else?
Any help would be greatly appreciated.
Kind regards,
Sasa
25.01.2022 15:51, je Guido Cioni napisal
Sasa, please take time to understand the errors before posting a full log here: the problems that you're getting are not because of this library but it seems to me to a poor understanding of Python and the environment where you're running the scripts. So it doesn't make sense to jump into a cronjob if you're not sure beforehand that you understand all the parts that are called inside.
As described in the README there are a (1) download/processing (2) plotting and (3) upload step.
Try to investigate all these steps separately before running the whole script again.
It seems that the download for you is working, although I'm not sure what this means
UNCHANGED_RECORD=0
But assuming the download and processing are working fine just try to run a single python script, for example python plot_cape.py with the flag debug=True, before running all of them at the same time.
Now focus on the error that you're getting in utils.py, it is coming from this line
If you do this os.environ[' '] you will get an error as there is no environment variable matching that name.
The apiKey value is used to make a dark background in some plots (plot_background_mapbox) and to find coordinate for cities to plot meteograms (get_city_coordinates). As I said, if you don't need these capabilities just comment out the lines using those functions. If you never use plot_meteogram.py you'll never use that key, same if you never use plotting routines that use plot_background_mapbox.
An alternative would be to get an account to mapbox so that you can get an API key.
guidocioni
commented
2 years ago Hello, I appreciate your time and consideration. I admin that my poor knowledge of Python affected the problems with the operation of the script. Yesterday I manage to successfully generate some images from script. I noticed that some products are "blank" (only borders and legend) or visually unreadable. Is this related to input data or something else? Any help would be greatly appreciated. Kind regards, Sasa 25.01.2022 15:51, je Guido Cioni napisal … Sasa, please take time to understand the errors before posting a full log here: the problems that you're getting are not because of this library but it seems to me to a poor understanding of Python and the environment where you're running the scripts. So it doesn't make sense to jump into a cronjob if you're not sure beforehand that you understand all the parts that are called inside. As described in the README there are a (1) download/processing (2) plotting and (3) upload step. Try to investigate all these steps separately before running the whole script again. It seems that the download for you is working, although I'm not sure what this means UNCHANGED_RECORD=0 But assuming the download and processing are working fine just try to run a single python script, for example python plot_cape.py with the flag debug=True, before running all of them at the same time. Now focus on the error that you're getting in utils.py, it is coming from this line https://github.com/guidocioni/icon_d2/blob/8d16e8a9215042c1f698c9a397bb3ace638bc97b/plotting/utils.py#L28 If you do this os.environ[' '] you will get an error as there is no environment variable matching that name. The apiKey value is used to make a dark background in some plots (plot_background_mapbox) and to find coordinate for cities to plot meteograms (get_city_coordinates). As I said, if you don't need these capabilities just comment out the lines using those functions. If you never use plot_meteogram.py you'll never use that key, same if you never use plotting routines that use plot_background_mapbox. An alternative would be to get an account to mapbox so that you can get an API key. -- Reply to this email directly, view it on GitHub [1], or unsubscribe [2]. Triage notifications on the go with GitHub Mobile for iOS [3] or Android [4]. You are receiving this because you authored the thread.Message ID: @.> -- Saša Zidar s.p. Vilharjeva cesta 12 6250 Ilirska Bistrica T: +386 51 654 010 E: @. W: www.meteolab.si [5] W: www.sasazidar.com [6] Links: ------ [1] #1 (comment) [2] https://github.com/notifications/unsubscribe-auth/AXONSVMI4OGK5IMGX2CORLTUX22HNANCNFSM5MWS7SNA [3] https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 [4] https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub [5] http://www.meteolab.si [6] http://www.sasazidar.com
could be everything...can you post an image example and the code used?
meteo1987
commented
2 years ago Here are some examples of generated maps. The files and code in plotting directory was not changed.
The only difference is in copy_data.run in the export mapbox API key and FTP connection information.
`#!/bin/bash
export MAPBOX_KEY="api-key"
cd "$(dirname "$0")";
export MODEL_DATA_FOLDER="/tmp/icon-d2/" export HOME_FOLDER=$(pwd) export N_CONCUR_PROCESSES=8 export NCFTP_BOOKMARK="icon-d2"
FTPU="ftp-user" FTPP="ftp-password" FTPS="ftp-host"
DATA_DOWNLOAD=true DATA_PLOTTING=true DATA_UPLOAD=true`
guidocioni
commented
2 years ago I can already see what the problem is here, but don't really understand how you're getting it if you didn't touch any code in the plotting routines.
If you look closely at the plots you can see that the values of pressure have the wrong units. For example here
the values are like 103062 and 103081. I assume these are Pascal and not HectoPascal (hPa), as the script expects. That's why you see a lot of lines, which cover basically all the other elements of the plot, as most of the scripts are supposed to draw contour lines with 2 to 5 units intervals, which are too many lines in case of Pascal!
Coming to the solution...I really don't get why the conversion is not done, as you're using the same code. Probably you're getting some error/warning and you didn't take care of solving it?
Just FYI the conversion is usually done using metpy/pint integration (more here https://unidata.github.io/MetPy/latest/tutorials/unit_tutorial.html), for example in this line
So you need to make sure the data after the initial preprocessing are in hPa. After that I guess everything will look normal.
meteo1987
commented
2 years ago I try everything but still the image output is the same. I think that the problem is in versions of libraries, Anaconda, Python, etc. Could you please tell me which version of libraries did you used for this scripts?
guidocioni
commented
2 years ago The problem is not related to the package versions...it is due to the fact that the input array is not in the right units. Did you try to debug the problem step by step? Like for example in a notebook just have a look at the array that is then sent to the plotting routine. Giving you the Python versions will not solve your problem..I update quite frequently
meteo1987
commented
2 years ago I manage to solve a few mysteries of wrong units on the map. In code I had to change this:
dset['2t'].metpy.convert_units('degC')
change to
dset['2t'] = dset['2t'].metpy.convert_units('degC')and this:
levels_mslp = np.arange(dset['prmsl'].min().astype("int"),
dset['prmsl'].max().astype("int"), 4.)
change to:
levels_mslp = np.arange(np.array(dset['prmsl']).min().astype("int"),
np.array(dset['prmsl']).max().astype("int"), 4.)Then the map was generated correctly.
Right now I am facing the problems with time...the T_min script is taking almost 7 hours to complete. Also, some scripts are not completed at all (CAPE, vmax_10m, gph_t_850 and gph_t_500, tmin, dBZ_cmax, WW, relhum_2m). Is the problem in the resources of my virtual PC?
guidocioni
commented
2 years ago I manage to solve a few mysteries of wrong units on the map. In code I had to change this:
dset['2t'].metpy.convert_units('degC') change to dset['2t'] = dset['2t'].metpy.convert_units('degC')and this:
levels_mslp = np.arange(dset['prmsl'].min().astype("int"), dset['prmsl'].max().astype("int"), 4.) change to: levels_mslp = np.arange(np.array(dset['prmsl']).min().astype("int"), np.array(dset['prmsl']).max().astype("int"), 4.)Then the map was generated correctly.
It seems that you're using a version of metpy where they updated the behaviour of the convert_units function to return a values instead than modifying in place. Anyway these kind of issues can always be solved by looking at the array before. You can also do the conversion yourself by doing e.g. dset['prmsl'] / 100. .
Right now I am facing the problems with time...the T_min script is taking almost 7 hours to complete. Also, some scripts are not completed at all (CAPE, vmax_10m, gph_t_850 and gph_t_500, tmin, dBZ_cmax, WW, relhum_2m). Is the problem in the resources of my virtual PC?
Yeah I think the problem are the resources. The plotting scripts are memory intensive as matplotlib is not really optimized for parallelized plotting. You can tune the number of processes that are spawned for plotting (see the README) to avoid running out of memory. 7 hours is still way too much...
Did you try to run only 1 one of the script ALONE without spawning the others? How long does it take in this case? Keep in mind that the one about plotting the max/min temperature takes quite a long time because of all the text that it needs to write on the plot. Still on our machine it takes about 1/2 minutes to complete.
meteo1987
commented
2 years ago Like you suggested, today I try to run only the cape_plot.py. After 3 hours still nothing. The script is still under processing, the last line on screen is:
Python 3.9.7 home/user/.anaconda/bin/python: can't find 'main' module in /tmp/icon-d2/'
guidocioni
commented
2 years ago Like you suggested, today I try to run only the cape_plot.py. After 3 hours still nothing. The script is still under processing, the last line on screen is:
`-------------------------------------------------------------------------
Icon-d2: Starting plotting of data - Wed 23 Feb 2022 15:15:23 CET
This is weird....
Python 3.9.7 home/user/.anaconda/bin/python: can't find 'main' module in /tmp/icon-d2/'
I've never seen this error before.
But I'm still seeing that you are launching the script from the parent one, copy_data.run. Did you try to execute it just from python, i.e. python plot_cape.py ? Did you try to do that while turning one the flag debug=True in the beginning, as I advised few messages ago? How long does it take to produce just this plot? If you defined all the variables correctly in utils.py you can just launch the plotting script DIRECTLY from the plotting folder to try it out!
The error message that you got is something that I cannot understand but in the past I had issues with the python script being stuck indefinitely before the plotting routine was launched. However, this was due to the fact that I was still passing chunked datasets and they were not loaded into memory. This line
should avoid that, so I would exclude that this is your problem.
meteo1987
commented
2 years ago Yesterday I try to execute the script from python python plot_cape.py. This is the log when I exclude dset = dset.drop(['lon', 'lat']).load() :
(base) user@user-VirtualBox:~/icon_d2-master/plotting$ python plot_cape.py plot_cape.py : Starting script to plot cape_cin plot_cape.py : Projection not defined, falling back to default (de) /home/user/.anaconda/lib/python3.9/site-packages/xarray/core/indexing.py:1227: PerformanceWarning: Slicing is producing a large chunk. To accept the large chunk and silence this warning, set the option
with dask.config.set(**{'array.slicing.split_large_chunks': False}): ... array[indexer]
To avoid creating the large chunks, set the option
with dask.config.set({'array.slicing.split_large_chunks': True}): ... array[indexer] return self.array[key] /home/user/.anaconda/lib/python3.9/site-packages/xarray/core/indexing.py:1227: PerformanceWarning: Slicing is producing a large chunk. To accept the large chunk and silence this warning, set the option with dask.config.set({'array.slicing.split_large_chunks': False}): ... array[indexer]
To avoid creating the large chunks, set the option
with dask.config.set({'array.slicing.split_large_chunks': True}): ... array[indexer] return self.array[key] /home/user/.anaconda/lib/python3.9/site-packages/xarray/core/indexing.py:1227: PerformanceWarning: Slicing is producing a large chunk. To accept the large chunk and silence this warning, set the option with dask.config.set({'array.slicing.split_large_chunks': False}): ... array[indexer]
To avoid creating the large chunks, set the option
with dask.config.set(**{'array.slicing.split_large_chunks': True}): ... array[indexer] return self.array[key] plot_cape.py : Pre-processing finished, launching plotting scripts /home/user/.anaconda/lib/python3.9/site-packages/matplotlib/contour.py:1430: UserWarning: Warning: converting a masked element to nan. self.zmax = float(z.max()) /home/user/.anaconda/lib/python3.9/site-packages/matplotlib/contour.py:1431: UserWarning: Warning: converting a masked element to nan. self.zmin = float(z.min()) /home/user/.anaconda/lib/python3.9/site-packages/matplotlib/contour.py:1430: UserWarning: Warning: converting a masked element to nan. self.zmax = float(z.max()) /home/user/.anaconda/lib/python3.9/site-packages/matplotlib/contour.py:1431: UserWarning: Warning: converting a masked element to nan. self.zmin = float(z.min()) /home/user/.anaconda/lib/python3.9/site-packages/matplotlib/contour.py:1430: UserWarning: Warning: converting a masked element to nan. self.zmax = float(z.max()) /home/user/.anaconda/lib/python3.9/site-packages/matplotlib/contour.py:1431: UserWarning: Warning: converting a masked element to nan. self.zmin = float(z.min()) /home/user/.anaconda/lib/python3.9/site-packages/matplotlib/contour.py:1430: UserWarning: Warning: converting a masked element to nan. self.zmax = float(z.max()) /home/user/.anaconda/lib/python3.9/site-packages/matplotlib/contour.py:1431: UserWarning: Warning: converting a masked element to nan. self.zmin = float(z.min()) /home/user/.anaconda/lib/python3.9/site-packages/matplotlib/contour.py:1430: UserWarning: Warning: converting a masked element to nan. self.zmax = float(z.max()) /home/user/.anaconda/lib/python3.9/site-packages/matplotlib/contour.py:1431: UserWarning: Warning: converting a masked element to nan. self.zmin = float(z.min())`
After 9 hours I canceled the script execution.
Then I run the script with included dset = dset.drop(['lon', 'lat']).load() . Here is the result:
base) user@user-VirtualBox:~/icon_d2-master/plotting$ python plot_cape.py plot_cape.py : Starting script to plot cape_cin plot_cape.py : Projection not defined, falling back to default (de) /home/user/.anaconda/lib/python3.9/site-packages/xarray/core/indexing.py:1227: PerformanceWarning: Slicing is producing a large chunk. To accept the large chunk and silence this warning, set the option
with dask.config.set(**{'array.slicing.split_large_chunks': False}): ... array[indexer]
To avoid creating the large chunks, set the option
with dask.config.set({'array.slicing.split_large_chunks': True}): ... array[indexer] return self.array[key] /home/user/.anaconda/lib/python3.9/site-packages/xarray/core/indexing.py:1227: PerformanceWarning: Slicing is producing a large chunk. To accept the large chunk and silence this warning, set the option with dask.config.set({'array.slicing.split_large_chunks': False}): ... array[indexer]
To avoid creating the large chunks, set the option
with dask.config.set({'array.slicing.split_large_chunks': True}): ... array[indexer] return self.array[key] /home/user/.anaconda/lib/python3.9/site-packages/xarray/core/indexing.py:1227: PerformanceWarning: Slicing is producing a large chunk. To accept the large chunk and silence this warning, set the option with dask.config.set({'array.slicing.split_large_chunks': False}): ... array[indexer]
To avoid creating the large chunks, set the option
with dask.config.set(**{'array.slicing.split_large_chunks': True}): ... array[indexer] return self.array[key] Killed
Next week I will install Ubuntu on standalone PC and try to run the scripts again. I think that the VirtualBox is not a good option for running this scripts?
guidocioni
commented
2 years ago Yes I think you would be better off with a standalone PC.
You need at least some decent amount of RAM to do parallel processing.
Still, with few GBs of RAM you should be able to just produce 1 single plot (that's why I suggested you to use the debug=True option).
I'll close this issue because it is getting too long and we deviated from the original question. In the future if you open a new issue please keep the description concise without attaching hundreds of lines from the log and have 1 issue per problem that you face.
When I execute cronjob, in file cron.log following errors appears:
Python 3.8.10 Traceback (most recent call last): File "plot_meteogram.py", line 9, in <module> from utils import * File "/tmp/icon-d2/utils.py", line 28, in <module> apiKey = os.environ['PUBLIC API-KEY] File "/usr/lib/python3.8/os.py", line 675, in __getitem__ raise KeyError(key) from None KeyError: 'PUBLIC API-KEY' Traceback (most recent call last): File "plot_cape.py", line 4, in <module> from utils import * File "/tmp/icon-d2/utils.py", line 28, in <module> apiKey = os.environ['PUBLIC API-KEY'] File "/usr/lib/python3.8/os.py", line 675, in __getitem__ raise KeyError(key) from None KeyError: 'PUBLIC API-KEY'...etcIs the problem in API-KEY or something else?