RaddishLabs / datasources

Access to all datasources that could be relevant for modeling. See Projects tab for more information
0 stars 0 forks source link

Dates of USDA annual reports for different crops #22

Open akudrya opened 6 years ago

akudrya commented 6 years ago

@AnnaOsipenko could you please help with identifying report dates for 2017 for major crops in US.

The major crops are: corn, soybean, rice, wheat, cotton, sugarcane.

Where to get the information:

Here's the source for reports http://usda.mannlib.cornell.edu/MannUsda/viewDocumentInfo.do?documentID=1194

Please open all monthly reports in pdf for 2017 (e.g. http://usda.mannlib.cornell.edu/usda/waob/wasde//2010s/2017/wasde-09-12-2017.pdf) and see whether the projections for these corps are forecasted or estimated. Based on it please specify the dates of issue for crop-specific reports for all the crops. Thanks

cc @katelengold

AnnaOsipenko commented 6 years ago

@akudrya I have checked all 12 reports and actually see that figures change from projected to estimated for the year 2016/17 for all the crops in one and the same report - for May, released on May 10th.

Previous scheme (for Jan - Apr): 2015/16 - estimated, 2016/17 - projected New scheme (starting from May and until the end of the year): 2016/17 - estimated, 2017/18 - projected

It seems the reports are not so crop-specific. Does it give us something? Or do I interpret anything incorrectly (I looked through the tables for both US and World)?

Reports also contain numerical assessments of projections reliability. Can this probably help?

katelengold commented 6 years ago

@AnnaOsipenko Does that mean that there are no deadlines when reports of actual yield are being published. I am not quite sure that's accurate -- the way our competitors position themselves is "predicted before USDA" so there must be some date there... Can you please double-check this week?

AnnaOsipenko commented 6 years ago

@katelengold Got your idea! What I see analysing the logic of publishing USDA reports and our competitors' materials:

1) The question is why the USDA guys don't stop "projecting" for the current year, say, in November or December when it's high time they knew the actual result, right? My thoughts are below (detailed description of what was happening in 2017).

2) To understand the meaning of this "predicted before USDA", I looked at the TellusLabs' Kernel use case for US corn in 2017 (https://telluslabs.com/blogs/news/telluslabs-coverage-2017-us-growing-season). Timeline: May: USDA says the yield will be 170.7 bu/ac June: USDA - 170.7 bu/ac. TL starts predicting and says the yield will be 166,2 bu/ac (only "green" model used) July: USDA - 170.7 bu/ac. TL switches on the second, "blue" model, both models still think yield will be somewhat less than 170 bu/ac August: USDA - 169.5 bu/ac. TL models in the course of August move closer to 172.5 - 175 bu/ac. On Aug 28 the "blue" model already shows the yield of more than 175 bu/ac September: USDA - 169.9 bu/ac. TL models are both above 175 bu/ac October: USDA - 171.8 bu/ac. TL projections are between somewhat 176 and nearly 180 bu/ac November: USDA finally adjusts its projections and says - 175.4 bu/ac. This is what Tellus "knew" in August-September and this is what they underline when they say they were two months faster.

2018-07-09 11 51 44

This is where the case description ended but let's trace a bit further. December: USDA - 175.4 bu/ac January 2018: USDA - 176.6 bu/ac. Again we can compare it with TL projections and see that they were near this figure already in October. February - April 2018: the USDA figure remains the same - 176.6 bu/ac. They still specify it as "projection" until May, when this figure moves to "estimation" of the past year, whereas projections start to be made for the new period. But can this be simply the report format observation? Perhaps, USDA simply keeps its tables unified until the new period forecasts become available. In this case we can say for sure that in January USDA finalises its estimation of the previous year and gives the actual result.

3) Following this logic, I will check when figures stop changing in the USDA reports for each crop and post another comment.

AnnaOsipenko commented 6 years ago

On point 3 of my previous comment:

Year 2017 Corn - final figure fixed in January '18 Soybean - final figure fixed in January '18 Rice - final figure fixed in January '18 Wheat - final figure fixed in October '17 (again, does it mean USDA was ready to count actual yield or did they simply predict right so that there was no further need in altering this figure?) Cotton - firstly, the figure was fixed in January '18, but then changed in March '18... Sugarcane - figures change, but here they describe production not yield projection / estimation so it seems quite logical.

katelengold commented 6 years ago

@AnnaOsipenko Very interesting! Thank you for this analysis. So basically they call it "projection" until the very end. What about geographies? How USDA reports cover different geographies and with what granularity? How does it compare to what Tellus delivers in the end?

AnnaOsipenko commented 6 years ago

@katelengold In USDA reports there are two types of estimations / projections: US and World. Granularity is always per country. Yield is covered only by the US part. All World tables contain figures on production instead of yield, and comparing these two things directly when forecasting is of course inaccurate. For that, when TL is competing with USDA, they always match speed only in the US part.

On the other hand, TL tries to predict yields in other parts of the world. In particular, in April they announced that this year their models will cover corn, soy and wheat crops in 8 countries (EU is considered a single country):

2018-07-11 11 57 48

However, the announcement contains a very important note: these trend estimates have no information from the current season. They are a reflection of long-term trends only. While it is convenient to compare these trend estimates to current government forecasts, they are different in kind. Thus, these predictions do not represent somewhat a fully competitive product for now, don't they?

katelengold commented 6 years ago

This is very interesting. Can we dive into that during our next call? I want to learn more about your findings and brainstorm together how we can fit this info into our positioning.

AnnaOsipenko commented 6 years ago

@katelengold Sure, with pleasure. I will try to have everything systematized.

AnnaOsipenko commented 6 years ago

I found an interesting note at the Gro Intelligence website: "Next month [August] is the first WASDE to provide a fully new estimate rather than another reprinting or minor modification of the current number". They name the August report critical. I think this should be taken into account.

katelengold commented 6 years ago

Very interesting. Thank you for this note, Anna!