fraymio / modis-tools

Tools for working with the MODIS API and MODIS data.
Apache License 2.0
23 stars 12 forks source link

Downloaded hdf files #36

Open RajasBharambe opened 1 year ago

RajasBharambe commented 1 year ago

Describe the bug I downloaded the hdf files using modis-tools, but unable to open the file: It says File Signature not found when I browse through some solutions it says, maybe the file is corrupted.

Your Environment:

iamchrisearle commented 1 year ago

Hi @RajasBharambe , can you share the code you used to download the files (omitting your login credentials) so that can try and replicated on my side.

RajasBharambe commented 1 year ago

Sure, here's the code:

from modis_tools.auth import ModisSession from modis_tools.resources import CollectionApi, GranuleApi from modis_tools.granule_handler import GranuleHandler

username = "" password = ""

session = ModisSession(username=username, password=password)

collection_client = CollectionApi(session=session) collections = collection_client.query(short_name="MOD11A1", version="061")

granule_client = GranuleApi.from_collection(collections[0], session=session)

india_bbox = [71.00, 29.00, 86.50, 18.00] granules = granule_client.query(start_date="", end_date="", bounding_box=india_bbox)

file_paths = GranuleHandler.download_from_granules(granules, session, ext = ("hdf", "h5", "nc", "xml"))

GranuleHandler.download_from_granules(granules, session=session, path="")

iamchrisearle commented 1 year ago

Can you provide the start and end dates that you used as well?

RajasBharambe commented 1 year ago

Sure, i want to download the data from 1990 to 2022

iamchrisearle commented 1 year ago

@RajasBharambe, I'm not able to recreate that error on my end using a subset of years for that product. Just a few things that might help you troubleshoot further: The bounding box you provided looks incorrect. The box should take the form [xmin, ymin, xmax, ymax]. You can try this box for India to see if that changes anything for you:

india_bbox = [68.17822, 6.755953, 97.41529, 35.67455]

Querying 32 years of data at a time raises some concerns as well. I recommend querying in smaller batches to ensure you have sufficient storage, bandwidth, etc. (I suspect this will reach terrabytes of data) and to ensure the data quality in older time frames works for your use case.

If you still get this error, please provide any changes to the code you are using as well.

RajasBharambe commented 1 year ago

Ohk @cearlefraym, I will try it, thanks for your suggestion. I haven't done any changes, but if I did, I would surely describe it,

iamchrisearle commented 1 year ago

Any update on this @RajasBharambe? If this is resolved I can close out the issue

RajasBharambe commented 1 year ago

No, its not. Im getting the same error after trying. Could you please tell me any alternative way ?

iamchrisearle commented 1 year ago

Can you post your most recent code used to reproduce the error?

RajasBharambe commented 1 year ago

Sure, the code is as follows:

from modis_tools.auth import ModisSession
from modis_tools.resources import CollectionApi, GranuleApi
from modis_tools.granule_handler import GranuleHandler

username =
password =

# Authenticate a session
session = ModisSession(username=username, password=password)

# Query the MODIS catalog for collections
collection_client = CollectionApi(session=session)
collections = collection_client.query(short_name="MOD11A1", version="061")

# Query the selected collection for granules
granule_client = GranuleApi.from_collection(collections[0], session=session)

# Filter the selected granules via spatial and temporal parameters
india_bbox = [55.00, 5.00, 105.00, 45.00]
india_granules = granule_client.query(start_date="2020-01-01", end_date="2023-01-01", bounding_box=india_bbox)

# Download the granules
GranuleHandler.download_from_granules(india_granules, session)
RajasBharambe commented 1 year ago

Sir, please tell me what should I do ?

iamchrisearle commented 1 year ago

The code you posted above is working for me, although it takes a while to start up as you are querying 3 years of data which is over 31000 files. image

I can only recommend you double check your login credentials, restart your kernel to make sure you don't have out-dated variables in memory for your bounding box, and chunk your time frame into smaller periods to avoid throttling, long download times, disk space issues etc.

RajasBharambe commented 1 year ago

Yes, I tried doing all the possible ways you mentioned, but same error is occuring.

image

iamchrisearle commented 1 year ago

Can you send the full stack trace?

iamchrisearle commented 11 months ago

Are you still getting this error? And if so, can you please provide a stack trace along with your code to help troubleshoot further?

RajasBharambe commented 11 months ago

Yes I am getting the same error, after many hits and trials

image