Open tomconte opened 1 year ago
TomAugspurger
commented
1 year ago Unfortunately, the USGS servers returned some errors for older scenes, and the text of those errors were written to blob storage.
See https://github.com/microsoft/PlanetaryComputer/discussions/101 for some discussion. I'll post there once it's fully resolved.
tomconte
commented
1 year ago Thanks Tom! As far as I can tell, we have executed our aggregation process successfully several times in the past, latest on February 1st. Has the data changed since that date? Just trying to figure out why we haven't seen that error before.
scottyhq
commented
1 year ago Figured I'd add another example since I've also encountered missing data issues. My understanding of the discussion in #101 is that some scenes are fixed on the USGS side and others are not... Here is a scene that has a valid preview and STAC, but the assets are corrupted, so when you ultimately try to read assets you'll get RasterioIOError: not recognized as a supported file format :
I think this illustrates another example of where some sort of metadata and data versioning would be useful to also have in the STAC metadata (https://github.com/microsoft/PlanetaryComputer/discussions/124).
TomAugspurger
commented
1 year ago I think this illustrates another example of where some sort of metadata and data versioning would be useful to also have in the STAC metadata (https://github.com/microsoft/PlanetaryComputer/discussions/124).
Yeah, we've sketched out a design for implementing the version extension but it'll be a decent amount of work to implement.
In the meantime, we've started doing the basics (computing and checking hashes before uploading files) and just need to clean this up for the historical data.
Has the data changed since that date? Just trying to figure out why we haven't seen that error before.
The Blob Storage service does provide a last modified property, which shows this asset hasn't changed since November 2022.
In [1]: import azure.storage.blob
In [2]: import planetary_computer
In [3]: cc = planetary_computer.get_container_client("landsateuwest", "landsat-c2")
In [5]: dict(cc.get_blob_client("level-2/standard/oli-tirs/2022/047/028/LC08_L2SP_047028_20221108_20221115_02_T1/LC08_L2SP_047028_20221108_20221115_02_T1_SR_B4.TIF").get_blob_properties())
Out[5]:
{'name': 'level-2/standard/oli-tirs/2022/047/028/LC08_L2SP_047028_20221108_20221115_02_T1/LC08_L2SP_047028_20221108_20221115_02_T1_SR_B4.TIF',
'container': 'landsat-c2',
'snapshot': None,
'version_id': None,
'is_current_version': None,
'blob_type': <BlobType.BlockBlob: 'BlockBlob'>,
'metadata': {},
'encrypted_metadata': None,
'last_modified': datetime.datetime(2022, 11, 17, 8, 22, 30, tzinfo=datetime.timezone.utc),
'etag': '"0x8DAC874DEA18EC9"',
'size': 14407,
'content_range': None,
'append_blob_committed_block_count': None,
'is_append_blob_sealed': None,
'page_blob_sequence_number': None,
'server_encrypted': True,
'copy': {'id': None, 'source': None, 'status': None, 'progress': None, 'completion_time': None, 'status_description': None, 'incremental_copy': None, 'destination_snapshot': None},
'content_settings': {'content_type': 'image/tiff', 'content_encoding': None, 'content_language': None, 'content_md5': bytearray(b'\x10\xb4\x15\xbfY\xe3\xdf\xe7-\xd7\x90w\x0c\xcd\x1e\x92'), 'content_disposition': None, 'cache_control': None},
'lease': {'status': 'unlocked', 'state': 'available', 'duration': None},
'blob_tier': 'Hot',
'rehydrate_priority': None,
'blob_tier_change_time': datetime.datetime(2022, 11, 17, 8, 22, 30, tzinfo=datetime.timezone.utc),
'blob_tier_inferred': None,
'deleted': False,
'deleted_time': None,
'remaining_retention_days': None,
'creation_time': datetime.datetime(2022, 11, 17, 8, 22, 30, tzinfo=datetime.timezone.utc),
'archive_status': None,
'encryption_key_sha256': None,
'encryption_scope': None,
'request_server_encrypted': True,
'object_replication_source_properties': [],
'object_replication_destination_policy': None,
'last_accessed_on': None,
'tag_count': None,
'tags': None,
'immutability_policy': {'expiry_time': None, 'policy_mode': None},
'has_legal_hold': None,
'has_versions_only': None}
Hi! Is there a problem with the Landsat asset below? When I try to download it locally, I do get a file with a .tiff extension, but it looks like it contains some HTML? (edit: removed SAS token)
https://landsateuwest.blob.core.windows.net/landsat-c2/level-2/standard/oli-tirs/2022/047/028/LC08_L2SP_047028_20221108_20221115_02_T1/LC08_L2SP_047028_20221108_20221115_02_T1_SR_B4.TIF