search

pik-piam / mrcommons

GNU Lesser General Public License v3.0
1 stars 30 forks source link

Bug: Many filenames (links) in downloadFAO_online.R are outdated. #62

Closed moulai closed 2 years ago

moulai commented 2 years ago

Many filenames (links) in downloadFAO_online.R are outdated.

For example, the file name of Crop is Production_Crops_E_All_Data_(Normalized).zip, however, this file cannot be found in the latest datasets_E.xml (downloaded from: http://fenixservices.fao.org/faostat/static/bulkdownloads/datasets_E.xml).

And the link http://fenixservices.fao.org/faostat/static/bulkdownloads/Production_Crops_E_All_Data_(Normalized).zip has also expired.

By looking at the FAO website (https://www.fao.org/faostat/en/#data/QCL), it can be found that there is no Crops under the Production category, but Crops and livestock products, which should be downloaded from https://fenixservices.fao.org/faostat/static/bulkdownloads/Production_Crops_Livestock_E_All_Data_(Normalized).zip.

Hope that downloadFAO_online.R can be updated, after all, the FAO_online source is used a lot.

In addition, must some functions require the use of Production_Crops_E_All_Data_(Normalized).zip? Now that the file is not open to be downloaded from FAO, does it mean that some functions may not be available?

Thank you for your kind help and reply!

@dklein-pik @johanneskoch94 @caviddhen @tscheypidi @giannou

moulai commented 2 years ago

In addition, the file Fodder.csv cannot be downloaded directly from FAO.

I made Fodder.csv myself according to the code comments, but I am not sure that the file I made is what the program needs.

When running readFAO("Fodder"), I get an error: "Country" column does not exist.

I would appreciate it if you could provide the file Fodder.csv!

Attached is the Fodder.csv that I made myself (which will give an error when read it). Fodder.csv

PS: The code comments on Fodder in readFAO.R are as follows.

Fodder (data that has been manually downloaded from the FAOSTAT website as
seperate .xls files via a search for "forage" and "fodder" withing
Production-Crops. These datasets have been added together to a "Fodder.csv" file)
moulai commented 2 years ago

I checked the download links corresponding to each file and found that the following files in downloadFAO_online.R have expired:

CBCrop
CBLive
Crop
CropProc
EmisAgBurnSavanna
EmisAgCultOrgSoil
EmisAgTotal
EmisLuBurnBiomass
EmisLuCrop
EmisLuForest
EmisLuGrass
EmisLuTotal
FSCrop
FSLive
LiveHead
LivePrim
LiveProc
caviddhen commented 2 years ago

Hi there @moulai Thank you for flagging this !

I will try to update the download function in the next days.

Regarding fodder, unfortunately we can not share our internal file as per communication with FAOSTAT but I'll compare ours to the one you've constructed and let you know.

moulai commented 2 years ago

Thank you for your kind reply!

I can totally understand your inconvenience to share internal files about Fodder. Thank you very much for your kindness to help compare your files with mine.

Also, for reference, I have tried to search the current datasets_E.xml file for filenames related to or similar to the unavailable files. But there are still many files that are missing. The result is as follows:

CBCrop: FoodBalanceSheets_E_All_Data_(Normalized).zip, FoodBalanceSheetsHistoric_E_All_Data_(Normalized).zip
CBLive: CommodityBalances_(non-food)_(2010-)_E_All_Data_(Normalized).zip
Crop: Production_Crops_Livestock_E_All_Data_(Normalized).zip
CropProc: not found
EmisAgBurnSavanna: not found
EmisAgCultOrgSoil: Emissions_Drained_Organic_Soils_E_All_Data_(Normalized).zip
EmisAgTotal: not found
EmisLuBurnBiomass: Emissions_Land_Use_Fires_E_All_Data_(Normalized).zip
EmisLuCrop: not found
EmisLuForest: Emissions_Land_Use_Forests_E_All_Data_(Normalized).zip
EmisLuGrass: not found
EmisLuTotal: not found
FSCrop: SUA_Crops_Livestock_E_All_Data_(Normalized).zip
FSLive: not found
LiveHead: Production_Crops_Livestock_E_All_Data_(Normalized).zip
LivePrim: not found
LiveProc: not found
caviddhen commented 2 years ago

Dear Mou, After having looked into the FAOSTAT update, we are in communication with FAOSTAT for a more long-lasting solution to their data updates, but unfortunately at the moment we do not have the capacity to update the data links.

You may be able to adjust the readFAO_online function in your personal branch to account for some of the missing files, for instance it seems like CropProc LivePrim LiveProc have all been merged into Production_Crops_Livestock_E_All_Data_(Normalized).zip. Similarly with FSLive into SUA_Crops_Livestock_E_All_Data_(Normalized).zip. With the emissions datasets I'm not so sure.

And unfortunately also the Fodder dataset used to be available on FAOSTAT but is no longer - we also do not have permission to share it at the moment, but will update you if anything changes, we've gotten in touch with FAOSTAT.

moulai commented 2 years ago

Dear Chen, Thanks for your reply and effort to resolve this issue. Thank you for your suggestion. I will try to make some modifications in my personal branch. I look forward to FAOSTAT agreeing to release these data in the future.