Issues related to product version update in #238

[ranghetti]

I have updated MODIStsp_prodopts.xml.zip with MCD64A1 and deleted old .RData and launched load_prodops. MODIStsp_prodopts.RData is properly updated.

But: I get "Warning: Error in MODIStsp_get_prodlayers: Version 006 is not available for product Burned_Monthly_500m (MCD64A1)." Modistsp seems to assume that 006 should be available for all products. Something else needs to be fixed.

006 has been depreciated for almost all products (even the download folders have been deleted), but MODIStsp seems to be expecting "006" version product information to exist too. MODIStsp should be updated accordingly. 061 should be default.

If you fix this, I can update the .xml for all products.

Originally posted by @pkautio in https://github.com/ropensci/MODIStsp/issues/238#issuecomment-1676955787

[ranghetti]

@pkautio I have fixed that in https://github.com/ropensci/MODIStsp/commit/81c3c6b4ba25ac2c2407a8323f2d26b4a7aee8ed, please update MODIStsp from branch devel (and make PR to this branch). For any other issue, please open this thread.

[pkautio]

I updated MODIStsp to the latest development version. Now the error above has gone away, but when I try to download "061" version, I get: "Error in MODIStsp_get_prodlayers(o$selprod) : Version 006 is not available for product Burned_Monthly_500m (MCD64A1)"

There is still an assumption somewhere that "006" should exist.

"MODIStsp_get_prodlayers.R" line 31. Should the value be "061" instead of "006"?

[ranghetti]

I am correctly downloading and processing them. Please, post your MODIStsp_ProdOpts.xml.zip content (from tag <MCD64A1> to </MCD64A1>).

[ranghetti]

"MODIStsp_get_prodlayers.R" line 31. Should the value be "061" instead of "006"?

No, this is only the default version, but the selected version is passed to the function when called.

[pkautio]

MODIStsp(gui             = FALSE,
         out_folder      = "FireData",
         out_folder_mod  = "FireDataMod",
         selprod         = "Burned_Monthly_500m (MCD64A1)",
         prod_version    = "061",
         bandsel         = "Burn_Date",
         user            = "removed",
         password        = "removed",
         start_date      = "2001.01.01", 
         end_date        = "2023.12.31", 
         verbose         = TRUE,
         spatmeth        = "file",
         spafile         = spatial_filepath,
         out_format      = "GTiff")

GDAL version in use: 3.5.2
Error in MODIStsp_get_prodlayers(o$selprod) : 
  Version 006 is not available for product Burned_Monthly_500m (MCD64A1).
If you think this is an error, please open a new GitHub issue at 'https://github.com/ropensci/MODIStsp/issues' and report it.

[pkautio]

I basically updated all "006"-related references to "061" within MCD64A1:

                <v_number>061</v_number>
                <prod_fullname>MCD64A1: Burned Area Monthly L3 Global 500m V061</prod_fullname>
                <main_out_folder>Burned_Monthly_500m_v61</main_out_folder>
                <native_res>463.3127</native_res>
                <http_terra>https://e4ftl01.cr.usgs.gov/MOTA/MCD64A1.061/</http_terra>
                <http_aqua>https://e4ftl01.cr.usgs.gov/MOTA/MCD64A1.061/</http_aqua>

It makes no sense to keep "006" default, since it is no longer available for any product except for MOD44W.

[ranghetti]

Ok, there was a bug fixed in 80efcb265c888a3dc8d4447f1d5a98b0ee3eb8b8. Now it may work also in manual mode.

I basically updated all "006"-related references to "061" within MCD64A1:

This is ok in products for which band structure did not change from version 006 to 061. For subsequent products, it is strategic to remember to check this assumption, by comparing layers from version 006 (e.g. https://lpdaac.usgs.gov/products/mcd64a1v006, section "Layers") to 061 (e.g. https://lpdaac.usgs.gov/products/mcd64a1v061/). You can find an example o diverging structures in the xml file, comparing code of the two versions of product MOD09A1.

It makes no sense to keep "006" default, since it is no longer available for any product except for MOD44W.

This is true, I changed that in f8ea3ac68a5e72ded66efa34655126135b3325e6. According to this change, tests will probably fail, so it will be necessary to test them before releasing the new master.

Thank you

[pkautio]

I tested .xml file with both "006" and "061" versions of data. Now it works - I can download "061" version of data. If I remove "006", I will get the error above.

Yes, I need to double-check all the layers too, but this was only to test the principle. The .xml file should work without any reference to "006".

[ranghetti]

I tested .xml file with both "006" and "061" versions of data. Now it works - I can download "061" version of data. If I remove "006", I will get the error above.

Do you mean if you remove <version> <v_number>061</v_number> ... </version> from the xml file? Actually I am leaving only 061 and all is working (using your chunk and changing only spatfile).

[hakimabdi]

I've modified the XML file as suggested here, but still cannot download v061 of MCD64A1. Any further suggestions how to remedy this?

<MCD64A1>
            <name>Burned_Monthly_500m (MCD64A1)</name> 
            <cat_01>Land Cover Characteristics</cat_01> 
            <cat_02>Thermal Anomalies and Fire</cat_02>
            <file_prefix_terra>MCD64A1</file_prefix_terra>
            <file_prefix_aqua>MCD64A1</file_prefix_aqua>
            <versions>
                <version>
                    <v_number>061</v_number>
                    <prod_fullname>MCD64A1: Burned Area Monthly L3 Global 500m V061</prod_fullname>
                    <main_out_folder>Burned_Monthly_500m_v61</main_out_folder>
                    <native_res>463.3127</native_res>
                    <http_terra>https://e4ftl01.cr.usgs.gov/MOTA/MCD64A1.061/</http_terra>
                    <http_aqua>https://e4ftl01.cr.usgs.gov/MOTA/MCD64A1.061/</http_aqua>
                    <combined>1</combined>
                    <tiled>1</tiled>
                    <www>https://lpdaac.usgs.gov/products/mcd64a1v061</www>

The command I tried was like this:

MODIStsp(gui = FALSE, out_folder = "C:/Fire", out_folder_mod = "C:/Fire", selprod = "Burned_Monthly_500m (MCD64A1)", prod_version = "061", sensor = "both", download_server = "https://e4ftl01.cr.usgs.gov/MOTA/MCD64A1.061/", bandsel = "Burn_Date", user = "user", password = "password", start_x = 18, end_x = 18, start_y = 4, end_y = 4, start_date = "2005-01-01", end_date = "2005-06-30", verbose = TRUE, spatmeth = "tiles", out_projsel = "User Defined", output_proj = "4326", out_res_sel = "Native", out_res = 463.3127, resampling = "near", reprocess = TRUE, delete_hdf = TRUE, nodata_change = FALSE, scale_val = FALSE, compress = "LZW", out_format = "GTiff")

And this is the error message I got:

GDAL version in use: 3.6.2 Error: assert_that: missing values present in assertion

There was no option to select version 061 in the GUI even after updating the XML file.

[pkautio]

@hakimabdi I got it working specifically with MCD64A1. You need to download the latest dev version of MODISTSP before the change.

I try to allocate time during next weekend to update all or at least most .xml entries.

[hakimabdi]

Ok, will do, thanks for your help!

[hamzamukhtar786]

Can anybody help me installation of this version. I don't know why i am getting error. Even i am following each step.

[ranghetti]

I try to allocate time during next weekend to update all or at least most .xml entries.

Thank you @pkautio, when you will have updated the xml please create a PR to the devel branch.

For the others, please do not use this thread for different issues. I am sorry ont to be available to provide assistance, as you would have seen. Thank you for your comprehension.

[pkautio]

I have now updated .xml file, which works fine in my environment. Please advice how I can update it?

[ranghetti]

If you have experience with GitHub you can open a PR to the develop branch; otherwise, attach here the file.

[pkautio]

Here is the full updated .xml version. I have not checked if there are some new products, but the existing one's should be up-to-date. All 060 versions are removed and 061 version added if it was missing.

I personally work with MCD64A1 and the downloading of data was tested with that product.

MODIStsp_ProdOpts.zip

[pkautio]

Pull request has been made including the .xml updates.

[vferng]

Dear developers and community,

I am trying to download MCD64A1 data (version 061) - see the code below- and I get similar errors (Error in if (prod_opts$tiled == 0) { : argument is of length zero).

MODIStsp(gui = FALSE, out_folder = 'MODIS', out_folder_mod = 'MODIS', selprod = 'Burned_Monthly_500m (MCD64A1)', prod_version = '061', bandsel = 'Burn_Date', sensor = 'both', user = 'user' , password = 'password', start_date = '2022.01.01', end_date = '2022.12.31', verbose = TRUE, bbox = c(18, -36, 55, -10), spatmeth = 'bbox', out_format = 'GTiff', compress = 'GTiff', out_projsel = 'User Defined', output_proj = '+proj=longlat +ellps=WGS84 +datum=WGS84', delete_hdf = TRUE, parallel = TRUE )

I've replaced my MODIStsp_ProdOpts.zip file by the one provided here, and also updated the version and still gives me back the same error. Do you know some solution?

Best wishes

[pkautio]

Hi,

1) Install latest development version of modiststp: install_github("ropensci/MODIStsp", ref="devel") 2) Delete file "C:\Users\%user%\AppData\Local\R\win-library\4.3\MODIStsp\ExtData\MODIStsp_ProdOpts.RData" 3) Replace file "C:\Users\%user%\AppData\Local\R\win-library\4.3\MODIStsp\ExtData\MODIStsp_ProdOpts.xml.zip" 4) Start MODIStsp and it downloads the data from xml file. Otherwise it will use the configuration from .RData file and that's the old content (not working)

[vferng]

Thank you so much!

[ranghetti]

Thank you @pkautio , I will update the package as soon as possible.

[ranghetti]

Fixed in github master version 2.0.10.9000 (6a9eba2ef86ba0d361d02407211b2c609859150f).