Tomorrows met data

[HamishBrownPFR]

@hol353 @hol430 @hut104 @junqi108 we are working on the grapevine model and bud burst is quite sensitive to short periods of chilling and are needing a more detailed daily air temperature interpolation. The problem is the approach we want to use needs to know tomorrows minimum temperature. APSIM currently has no notion of tomorrows weather.

I can program another a MinTtomorrow variable into IWeather and have it go get this from the met file. However, this feels quite messy and I am wondering if anyone can think of a better way.

[hut104]

I assume that you are using the parton model for diurnal temperature. We would need to work through a use-cases perhaps. We use diurnal temperatures currently to:

Calculating TT in plant model - need to iterate through the day Calculating VPD in microclimate - need to iterate through the day Calculating soil temperatures - not sure, but may need temperatures at different specific times

It may be that we could provide an array of hourly temperatures, or a method that returns the temperature for a time of day (0-24hours). We could do a similar thing for solar radiation for the photosynthesis models.

This would also allow us to provide tests of the model because we could have Pvs O diurnal temperature data in there.

---

From: Hamish Brown notifications@github.com Sent: Thursday, 7 May 2020 6:36 AM To: APSIMInitiative/ApsimX ApsimX@noreply.github.com Cc: Huth, Neil (A&F, Toowoomba) Neil.Huth@csiro.au; Mention mention@noreply.github.com Subject: [APSIMInitiative/ApsimX] Tomorrows met data (#5121)

@hol353https://github.com/hol353 @hol430https://github.com/hol430 @hut104https://github.com/hut104 @junqi108https://github.com/junqi108 we are working on the grapevine model and bud burst is quite sensitive to short periods of chilling and are needing a more detailed daily air temperature interpolation. The problem is the approach we want to use needs to know tomorrows minimum temperature. APSIM currently has no notion of tomorrows weather.

I can program another a MinTtomorrow variable into IWeather and have it go get this from the met file. However, this feels quite messy and I am wondering if anyone can think of a better way.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/APSIMInitiative/ApsimX/issues/5121, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AC2UVWSLYGYUO4NZQ7UDSZTRQHC3LANCNFSM4M2YJEPA.

[HamishBrownPFR]

@hut104 Take a look at the AirTemperatureFunction. I have already modified it to be able to use different interpolation methods. This can be used by any model, not just plant. Perhaps this needs to be generalized further so it can interpolate other met variables such as radiation. That would not be to hard to do. Would require a different interpolation method to what is used for temperature but I have structured it so you can change in different interpolation methods. I have already set it up so it can return the average or sum of sub daily values. Perhaps we should generalise AirTemperaturefunction further to be a SubDailyIntepolationFunction. That would not be hard. @jbrider how are you doing the sub daily interpolations for the canopy photosynthesis modelling?

Anyway, we are implementing a 24 hour interpolation that uses a sign during the daylight period and exponential during the dark and needs to know tomorrows Tmin to do this. My question here is what is the best way to go about getting TminTomorrow.

[Keith-Pembleton]

There are a number of use cases where it would be handy to have APSIM look forward in the met data. For the simulations behind CropARM we needed to do this and had to come up with a work around that was very messy and slowed down the simulations. What ever happens here it should be expanded beyond the next day and also beyond temperature.

[HamishBrownPFR]

@Keith-Pembleton How much further ahead do you need to look?

[Keith-Pembleton]

For the CropARM sims it is 2 weeks in advance

[BrianCollinss]

It seems like a time-series problem. There are some simple time-series model (weather generators) that can be implemented assuming linear correlation between the successive days.

In terms of hourly interpolation of temperature and radiation, I have implemented something in here and here that might be a starting point for you.

[jbrider]

@HamishBrownPFR I'm not familiar enough with how it works to comment - I do know it doesn't look forward though. DCaPST repository - we're adding it into NextGen soon.

[HamishBrownPFR]

Thanks @BenAbabaei . The temperature interpolation model is the same as what is implemented in #5123 except @junqi108 has it using tomorrows MinT for the dark period at the end of the day. @jbrider I have modified the AirtemperatureFunciton in release to work with different interpolation methods so it can use this method or the 3hour interp approach. In line with the comments of @hut104 it would be worth generalizing this further so it interpolates other met data as well. I am happy to do this as my head is in that space at the moment. @BenAbabaei, are you happy if I move your radiation and temperature interpolations into this method?

[HamishBrownPFR]

@Keith-Pembleton @hol353 @hol430 I wonder if a GetFutureMetData(When, What) method might be the way to go. Give it arguments of which day and which variable and it returns that. Would then be useful for what Keith needs as well as what I need and would not add cruft to IWeather. I will have a go at implementing something and seek further comment

[BrianCollinss]

@HamishBrownPFR absolutely. Feel free to make modifications if needed.

[jbrider]

@HamishBrownPFR generic sounds good - we weren't trying to reinvent any temperature functions - I just haven't gone looking for existing functionality yet - the To Do list is getting impressive. I think GetFutureMetData is a good name for the function.

[hol430]

How would you implement this? I expect it would have to work differently depending on which IWeather implementation it's dealing with?

[hut104]

Hi all, I wonder if down the track we could move to a more fundamental solution. The current approach will see different models using different approaches to interpolate temperatures within the current time step, or at least duplicating the calculations. The solution may be as simple as allowing met to interpolate temperature within the day and models use that. e.g. a call such as Met.Temperature(time), Met.SolarRadiation(time). I think this will cater for the wider set of use cases, and it would also allow you to get yesterday's value. Remember that to interpolate temperatures at the beginning of the day you need yesterday's maximum.

I'd implement the best approach for interpolation there and then we slowly move the models over to use this rather than the old CERES code which is we know is less than ideal.

---

From: Hamish Brown notifications@github.com Sent: Friday, 8 May 2020 5:06 AM To: APSIMInitiative/ApsimX ApsimX@noreply.github.com Cc: Huth, Neil (A&F, Toowoomba) Neil.Huth@csiro.au; Mention mention@noreply.github.com Subject: Re: [APSIMInitiative/ApsimX] Tomorrows met data (#5121)

@Keith-Pembletonhttps://github.com/Keith-Pembleton @hol353https://github.com/hol353 @hol430https://github.com/hol430 I wonder if a GetFutureMetData(When, What) method might be the way to go. Give it arguments of which day and which variable and it returns that. Would then be useful for what Keith needs as well as what I need and would not add cruft to IWeather. I will have a go at implementing something and seek further comment

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/APSIMInitiative/ApsimX/issues/5121#issuecomment-625441643, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AC2UVWU2X7RPGTAJVETLE53RQMBE7ANCNFSM4M2YJEPA.

[hol430]

Perhaps IWeather could have a method called something like GetWeatherData(DateTime time) where you pass in a DateTime and it returns some struct or something which contains all data currently encapsulated by IWeather (temp, radiation, wing speed of a swallow, ...). This method could interpolate within the day if needed, depending on the time component of the DateTime and it could also look ahead (or behind) depending on the date component. I don't know if we'd be better off modifying IWeather and its implementations to work this way or having this as a separate component.

[zur003]

Another thing to consider is that there may be a need down the line to provide hourly (or at least something finer than daily) values for variables other than temperature. Erosion models, for example, don't work well with daily values, as it gives only a poor indication of rain or wind intensity. Ideally, we should have something that can provide values on a finer temporal scale (maybe even 10 minutes), using observed data if it's available, or doing some clever interpolation (or, rather, distribution fitting) when it's not. That's a big ask, and I wouldn't expect an implementation in the short term, but the design should keep that sort of need in mind.

[HamishBrownPFR]

Hi guys, thanks for all the comments, really useful. I think this can be broken into three separate issues

1. Implement @BenAbabaei code for radiation interpolation into the AirTemperatureFunction and call it something more generic to show its true function (ApplyAndAgregateFunctionsToSubDailyInterpolatedMetDataFunction). Maybe not call it that.
2. Implement a GetWeatherData(Date) method as @hol430 recommends.
3. Redesign the weather component to deliver daily or sub-daily data based on @hut104 and @zur003 comments a. Read sub daily data from met file if there is a time column. In a lot of cases we have this data but throw it away to get the dailyData APSIM needs. b. If no time column interpolate using best approach for each weather component c. Allow additional daily data in met file like MaxWindGustSpeed and RainfallDuration that might be needed for interpolation d. Come up with a design for passing subdaily met data from IWeather e. Deprecate the code out in functions where sub hourly interpolation is currently happening

I will deal to the first two to resolve this issue and open a separate issue the capture the third.

[HamishBrownPFR]

@hol353 @hol430. I have hit a snag. Stop laughing. The reading of met data simply increments through lines in the file reader. So it is not easy to go for past dates and each time you call it the row number is incremented so the met row gets out of sync with the clock date. Met data would be easier to work with if it were read it into a date indexed dataframe and we could then just slice out the bits we want when we want them. I see there is a .Net equivalent to pandas, Deedle, which might fit the bill. I would be keen to have a play with this to work out how well it would fit but am unsure how to do the plumbing so the package can be called in the APSIMX solution.
Firstly, do you think this approach is worth investigating for weather. Secondly, advice on how to get Deedle or other datatable equivelent into the APSIM solution would be appreciated

[HamishBrownPFR]

I have found Deedle in NuGet and installed but have got something wrong because when in try to put in a using Deedle; dirrective it can't find the namespace. Having a closer looke ad Deedle, I would probably work out how to drive it but it doesn't appear to be as flexible as pandas. Python is supported in .net I think? Would it be possible to install and reference python packages such as pandas in the ApsimX solution?

[hol353]

The reason it steps through data line by line is because it is quicker. Quite often we feed it 100 year weather files and only use the last few years. It can be quite slow to ready all 100 years unnecessarily so instead weather 'seeks' to the required date and steps line by line from there. You could continue with this approach. When your GetWeatherData(toDate) method wants to read a date it could do: reader.Seek(todate); read the line of weather data reader.Seek(clock.Today); // reset the reader back to correct date.

Option 2

No need to use Deedle. You get quite a lot of flexibility with DataTable (rows and columns of data) and DataView (filtering a datatable) both built into .net.

You could read the entire contents into a DataTable: DataTable table = reader.ToTable();

Your GetWeatherData(toDate) method could then create a DataView on the DataTable, filter the data and then return the row: DataView view = new DataView(table); view.RowFilter = 'Date = ' + toDate.ToString('yyyy-MM-dd'); return view[0]; // returns entire row of data

This assumes that all weather files have a 'Date' column. In old APSIM we allowed year and day and other combinations (year, month, day). This might be the case in new APSIM as well. If so then you're going to have to do this the slightly harder way of calculating the row index of the date you're after by getting the date of the first row of the weather data (DateTime firstDate = DataTableUtilities.GetDateFromRow(table.Rows[0]);) and then subtracting the toDate from the firstDate to get a row index.

Option 3

Do a variation on Option 2 and just return the entire DataView (or DataTable) to the caller rather than filtering to a single row. This gives ultimate flexibility to the caller as they can filter it however they want.

Option 4

Microsoft now has an inbuilt DataFrame like structure - https://devblogs.microsoft.com/dotnet/an-introduction-to-dataframe/. Might be worth looking in to.

[hut104]

OR we keep it really simple, and just read in data one line at a time, but remember yesterday's data and read tomorrow's. That will meet our current requirements and still be very quick. The interpolation therefore only needs the time within the day to do the interpolation. Should only need to change a few lines of code.

---

From: Dean Holzworth notifications@github.com Sent: Monday, 11 May 2020 7:30 AM To: APSIMInitiative/ApsimX ApsimX@noreply.github.com Cc: Huth, Neil (A&F, Toowoomba) Neil.Huth@csiro.au; Mention mention@noreply.github.com Subject: Re: [APSIMInitiative/ApsimX] Tomorrows met data (#5121)

The reason it steps through data line by line is because it is quicker. Quite often we feed it 100 year weather files and only use the last few years. It can be quite slow to ready all 100 years unnecessarily so instead weather 'seeks' to the required date and steps line by line from there. You could continue with this approach. When your GetWeatherData(toDate) method wants to read a date it could do: reader.Seek(todate); read the line of weather data reader.Seek(clock.Today); // reset the reader back to correct date.

Option 2

No need to use Deedle. You get quite a lot of flexibility with DataTable (rows and columns of data) and DataView (filtering a datatable) both built into .net.

You could read the entire contents into a DataTable: DataTable table = reader.ToTable();

Your GetWeatherData(toDate) method could then create a DataView on the DataTable, filter the data and then return the row: `DataView view = new DataView(table); view.RowFilter = 'Date = ' + toDate.ToString('yyyy-MM-dd'); return view[0]; // returns entire row of data'

This assumes that all weather files have a 'Date' column. In old APSIM we allowed year and day and other combinations (year, month, day). This might be the case in new APSIM as well. If so then you're going to have to do this the slightly harder way of calculating the row index of the date you're after by getting the date of the first row of the weather data (DateTime firstDate = DataTableUtilities.GetDateFromRow(table.Rows[0]);) and then subtracting the toDate from the firstDate to get a row index.

Option 3

Do a variation on Option 2 and just return the entire DataView (or DataTable) to the caller rather than filtering to a single row. This gives ultimate flexibility to the caller as they can filter it however they want.

Option 4

Microsoft now has an inbuilt DataFrame like structure - https://devblogs.microsoft.com/dotnet/an-introduction-to-dataframe/. Might be worth looking in to.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/APSIMInitiative/ApsimX/issues/5121#issuecomment-626392651, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AC2UVWVXXQKVEZNMBP7I7NLRQ4MGPANCNFSM4M2YJEPA.

[HamishBrownPFR]

Thanks @hol353. Option 1 should do the trick and won't take to much programming.
I have used DataTable a bit but I couldn't work out how to slice by row and column, had to look throug each row to find the correct key and then return the column wanted. It seamed inefficient. I am pretty sure a DataTable reader could only read in the data between simulation start and end datas from a longer weather file so I don't think that would be an issue. Anyway, I will try option 1 and hope that will work.