tsarkany commented 5 years ago

I am trying to read a HDF file using the cis.read_data() function in Python. import cis import iris from cis import read_data from mpl_toolkits.basemap import Basemap import matplotlib.pyplot as plt import numpy as np a=cis.read_data('MOD06_L2.A2008001.0000.061.2017287164540.hdf','Cloud_Effective_Radius_16')

When I run this an error comes up:

ProductPluginException: An error occurred retrieving data using the product MODIS_L2. Check that this is the correct product plugin for your chosen data. Exception was TypeError: slice indices must be integers or None or have an __index__ method.

The same thing happens when I put in 'MODIS_L2' explicity as the plugin in the cis.read_data() function.

Am I using the wrong plugin? If not, then I can't work out how to read the file.

duncanwp commented 5 years ago

Could you let me know which version of CIS and the other libraries you are using? Feel free to attach a print out of your environment.

tsarkany commented 5 years ago

Dear Duncan, I am not sure whether this is what you are looking for but I have attached the output of the printenv command on Linux. When I type in cis version it outputs Using CIS version: 1.7.1 (Stable)

If this is not the right information then let me know! Best wishes, Tasmin

From: Duncan Watson-Parris notifications@github.com Sent: 09 August 2019 17:49:56 To: cedadev/cis cis@noreply.github.com Cc: Tasmin Sarkany tasmin.sarkany@lincoln.ox.ac.uk; Author author@noreply.github.com Subject: Re: [cedadev/cis] Plugin not working for .hdf MODIS data (#22)

Could you let me know which version of CIS and the other libraries you are using? Feel free to attach a print out of your environment.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/cedadev/cis/issues/22?email_source=notifications&email_token=AM3S3X3NNSQK4W5A6QY37ODQDWN3JA5CNFSM4IKVVOEKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD37GINA#issuecomment-519988276, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AM3S3X5FYNMZMRJNLJQRFRTQDWN3JANCNFSM4IKVVOEA.

tsarkany commented 5 years ago

Printout environment.pdf

duncanwp commented 5 years ago

OK, that's a nice recent version of CIS. To get the versions of the other Python packages could you send the output of running the command: conda list. Thanks!

w-k-jones commented 5 years ago

Created a new python 3 environment with cis to test this, getting the same error. The problem is in MODIS_L2.__field_interpolate on line 247 of cis/data_io/products/MODIS.py. I believe that all the slice indices now need to be specifically int type due to changes in numpy. The full error output is as follows:

Traceback (most recent call last): File "/home/users/wkjones/miniconda2/envs/cis_test/lib/python3.7/site-packages/cis/data_io/products/AProduct.py", line 197, in get_data data = product_cls().create_data_object(filenames, variable) File "/home/users/wkjones/miniconda2/envs/cis_test/lib/python3.7/site-packages/cis/data_io/products/MODIS.py", line 299, in create_data_object coords = self._create_coord_list(filenames, variable) File "/home/users/wkjones/miniconda2/envs/cis_test/lib/python3.7/site-packages/cis/data_io/products/MODIS.py", line 269, in _create_coord_list lat_data = self.field_interpolate(sd_lat) if apply_interpolation else sd_lat File "/home/users/wkjones/miniconda2/envs/cis_test/lib/python3.7/site-packages/cis/data_io/products/MODIS.py", line 247, in field_interpolate float(factor)) + output[int(factor / 2):(-1 * factor):factor, :] TypeError: slice indices must be integers or None or have an index method

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "", line 1, in File "/home/users/wkjones/miniconda2/envs/cis_test/lib/python3.7/site-packages/cis/init.py", line 41, in read_data data_list = read_data_list(filenames, variable, product) File "/home/users/wkjones/miniconda2/envs/cis_test/lib/python3.7/site-packages/cis/init.py", line 70, in read_data_list return DataReader().read_data_list(file_set, variables, product, aliases) File "/home/users/wkjones/miniconda2/envs/cis_test/lib/python3.7/site-packages/cis/data_io/data_reader.py", line 90, in read_data_list var_data = self._get_data_func(filenames, variable, product) File "/home/users/wkjones/miniconda2/envs/cis_test/lib/python3.7/site-packages/cis/data_io/products/AProduct.py", line 203, in get_data % (product_cls.name, type(e).name, e.args[0]), e) cis.data_io.products.AProduct.ProductPluginException: An error occurred retrieving data using the product MODIS_L2. Check that this is the correct product plugin for your chosen data. Exception was TypeError: slice indices must be integers or None or have an index method.

w-k-jones commented 5 years ago

Ah yeah, more specifically it's because this value in the slicing [(-1 * factor / 2 + 1)] isn't converted to int type

tsarkany commented 5 years ago

This is the output of conda list

packages in environment at /home/linc4371/miniconda3:

#

adamcpovey commented 5 years ago

It looks like the easiest fix is to replace float division / with integer division // in those lines. Any obvious reason why that wouldn't work?

simonrp84 commented 5 years ago

That floors the number, doesn't it? If so, would that break compatibility with the previous functionality? i.e: If (-1 * factor / 2 + 1) = 0.5 then was the 0th or 1st element of the array selected? Doing (-1 * factor // 2 + 1) will always select the lower element.

w-k-jones commented 5 years ago

numpy would previously select index 0 for 0.5. The only problem is trying to find out if the value was floored or cast as int when indexing, as that changes the behaviour for negative values

adamcpovey commented 5 years ago

Trying it in my own code doesn't work as you end up with a 2030x1350 coordinate and a 2030x1354 data field. It appears some rounding was being done somewhere.

duncanwp commented 5 years ago

To be honest, I'm keen to remove this code entirely since there are a few nasty cases which I'm not convinced it's properly dealing with, this issue of the 4 'missing' coordinates being one of them.

This is a first go at making this work with the python-geotiepoints package, which is probably the best way to tackle this: https://github.com/duncanwp/cis_plugins/blob/master/modis_fixed.py

The MOD06 and MOD04 products seem to have slightly different structures though so it still isn't working as generally as I'd like.

cedadev / cis

Plugin not working for .hdf MODIS data #22

packages in environment at /home/linc4371/miniconda3:

Name Version Build Channel