ability to select gridded climate files from the UI

[JJguri]

@hol430 We have developed bestiapop a Python script (soon to become a package) for climate data extraction and processing. The package offers the possibility to select a range of grids (5 km × 5 km resolution) and years producing various types of output files: csv, MET and soon TSV and SQLite. I think a big improvement for APSIM would be to create the ability to read MET files just selecting the year period, latitude, and longitude from the UI. Also, it could help to create long factorial combinations using climate as a factor for a range of latitudes and longitudes in a given region. Would it be possible? The script documentation is here. Although the code downloads data from the SILO database, it could be applied to other climate data sources e.g. NASA POWER as was implemented in R using APSIM CRAN.

[zur003]

I've thought a little about getting APSIM to download weather data, and there are a couple of things to consider. Should the downloads be done in the GUI prior to a run (as is currently implemented for soils), or does the downloading become a part of model execution, or both? There are also several services that might be used as a source for the data. How should the data source be configured or selected by the user?

I should note that run-time downloading of weather (and soil?), given a location and time frame, might be useful in the context of the World Modelers project, or similar exercises, where runs are done over a range of locations.

[peter-devoil]

It would be useful to provide some sort of localised cache of downloaded data so that it can be re-used by other simulations. Additionally, sometimes that data is time sensitive (eg today's data supercedes yesterdays).

[JJguri]

bestiapop was configured to generate MET files in a given directory at the user's machine, however, it could be adapted to host the data in some cloud for future simulations that require the same grid.

[hol430]

Yeah I can add something to the user interface to do this. We don't really have any python infrastructure as we do for R, but for now maybe we just assume python(3), pip and git are on path and proceed from there.

I guess we would want the ability to choose an output location to which the .met files will be saved. Perhaps if this field is left blank, we just generate temp files and delete them once finished with them. This should allow for both use cases mentioned by Eric and Pete above.

[JJguri]

Yeah I can add something to the user interface to do this. We don't really have any python infrastructure as we do for R, but for now maybe we just assume python(3), pip and git are on path and proceed from there.

I guess we would want the ability to choose an output location to which the .met files will be saved. Perhaps if this field is left blank, we just generate temp files and delete them once finished with them. This should allow for both use cases mentioned by Eric and Pete above.

good, it means that users will be able to select lat, long, and time period from GUI, and then bestiapop will download the data automatically?

[hol430]

Yep, I have a prototype which does that and seems to be working ok, although it's hard for me to test because the bestiapop seems to download ~400MB per variable per year, and my internet connection at the home office is quite slow (e.g. I estimate the wheat example would take ~2 days to download the met data and about 10 seconds to run the simulation).

If we're putting this into the gui, it would be handy if there were some sort of progress indication. What would be ideal would be if you could add a command-line flag to bestiapop which, when present, causes bestiapop to print overall progress (e.g. as a %) and/or a status (e.g. downloading variable x for year y - unsure if this would be feasible with -m switch). Apsim could then parse the console output and display the progress in the UI, as we do for the CroptimizR.

[peter-devoil]

Ouch. Is there an opendap service with this data?

[hol430]

Well the weather downloader already included in apsim downloads the data from here:

string url = String.Format("https://www.longpaddock.qld.gov.au/cgi-bin/silo/DataDrillDataset.php?start={0:yyyyMMdd}&finish={1:yyyyMMdd}&lat={2}&lon={3}&format=apsim&username={4}&password=silo",
                            startDate, endDate, entryLatitude.Text, entryLongitude.Text, System.Net.WebUtility.UrlEncode(entryEmail.Text));

Unsure if this comes from the same data source as the AWS bucket used by bestiapop, but it seems to only download data for the desired grid point.

[peter-devoil]

It's the same data, but AWS has all of it in netcdf. It took me a day or two to get it as well.. Opendap makes it easier to get "slabs" of data instead of single points.

[zur003]

Check with @her123, who has done a lot of work with weather data for Digiscape and World Modelers. He has a DAP server running for this stuff. (He's been on leave, but should be back soon.)

[JJguri]

Yep, I have a prototype which does that and seems to be working ok, although it's hard for me to test because the bestiapop seems to download ~400MB per variable per year, and my internet connection at the home office is quite slow (e.g. I estimate the wheat example would take ~2 days to download the met data and about 10 seconds to run the simulation).

If we're putting this into the gui, it would be handy if there were some sort of progress indication. What would be ideal would be if you could add a command-line flag to bestiapop which, when present, causes bestiapop to print overall progress (e.g. as a %) and/or a status (e.g. downloading variable x for year y - unsure if this would be feasible with -m switch). Apsim could then parse the console output and display the progress in the UI, as we do for the CroptimizR.

You do not need to download the NetCDF files in your computer. bestiapop has an option that creates the MET file directly from the cloud. Although this is lower than create from the local disk, it is useful if you do not have enough space in your PC to store the files. Please check the Usage section in the GitHub repo:

• Generate MET output files from AWS S3 for radiation, minimum temperature, maximum temperature and rain at the daily step between 2015 and 2016

This is specifically the code to do that. I suggest trying with one single year and for a specific grid cell. python bestiapop.py -a generate-met-file -y "2015-2016" -c "radiation max_temp min_temp daily_rain" -lat "-41.15 -41.1" -lon "145.5 145.55" -o C:\some\output\folder\

Regarding the progress indication, bestiapop already have that visualization tool incorporated using the package tqdm when you are running it through Anaconda Prompt. Please have a look at the following example.

[hol430]

Thanks @JJguri - yeah that command is basically what I was running last week, and even if it doesn't save the netcdf data to disk, I still see a lot of network traffic, which I'm assuming is why it's taking so long on my pc (CPU usage is quite low, probably to be expected since I'm not using -m).

As for the progress, there is progress reporting for the individual factors (year, latitude, climate variables), but it might be convenient if there was an option to show overall progress (so that the APSIM UI can show something like "downloading met data: 27%"). It's not really that important - for now I should be able to get apsim to internally aggregate the individual progress reports.

[darkquasar]

Hi @hol430 we could arrange for a general progress bar, I shall create an issue to add it as an enhancement. On the other side, we could also package BestiaPop into an executable using PyInstaller, would this be easier to call from APSIM?

[hol430]

Thanks @darkquasar, general progress bar will be handy for us. As for PyInstaller - I haven't used this before, does it generate a platform-specific executable, or will it be a cross-platform solution? Sounds useful either way, and will make this feature more accessible to lots of users, who won't have git/python installed.

[mpandreucci]

@LauraGagresearch you might want to have a look at this?

[JJguri]

@hol430 We updated the package documentation and bestiapop now is working well and it is generating files from the SILO API. Could you please check if your prototype is working? If so, we would like to add this functionality (as a prototype) in the package documentation.

[hol430]

What is the status on this feature? @JJguri @darkquasar are you guys using it, or do you use the "download weather file" option already in the GUI? I only ask because I'm thinking about runtime dependencies for our new cloud-based CI server, and am wondering about python.

[JJguri]

@hol430 no worries we are not using it for the moment, but we would like to integrate it in the near future, so people can generate the gridded MET file directly from the GUI passing the bestiapop attributes (lat, lon, year range).

[darkquasar]

Hi guys indeed this feature is not required and redundant since APSIM already allows for this. Bestiapop can now be launched freely in the cloud using Binder and files/analytics performed there.