[x] If bbox across 2 tiles, no proper handling is done <- also s2 mask not working then...
[ ] s2_cloudmask = xr.concat([arr, CLOUD_MASK], dim=“band”, coords = “minimal”, compat = “override”, combine_attrs = “override”) Laut Python ist 'id' is not present in all datasets.
[ ] Google earth engine too many requests error handling:
requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://earthengine.googleapis.com/v1alpha/projects/earthengine-legacy/thumbnails/7c599f6afdb68fc3eecb1b44de5bcab0-ba614d408f9e9c00c7899813e3f7b01b:getPixels
[x] Google authentication changed <- just need to update earth engine api, thats all!
[ ] Edge case: What if for query no data found?.. eg. time interval too short.
[ ] What if Earth engine fails <- currently just puts nans in s2_mask.. <- changed this to put different mask values. David is looking into it.
30SVB0101 2018-02-13
31SGU2407 2018-02-21
32NRJ2406 2018-01-29
33KTA0515 2018-01-18
36JUQ1600 2018-02-18 & 23 & 28
[x] Wrong data downloaded:
Check 28PHQ0110 2019-03-20
29PKQ1504 2019-06-26
29RMN0313 2019-11-13
29SQS2325 2018-01-24
30RYR1709 2018-03-11
30SVC0722 2021-01-20
30SXA0801 2021-08-15
32SKD2302 2021-03-24 & 2021-03-29
33LTE2200 2019-11-19 & 2019-11-24
34PEA2200 2018-03-11
35JQH0111 2020-08-01
36JTS0502 2018-10-21 & 26
[x] Black boundary top & bottom just ~2 pixels:
28QED1917 2020-05-24
29RMN0710 2019-01-20
29RNP0107 2020-03-12
[ ] Found snow in AFrica :D
29SQR2209 2018-11-25
29SQR2425 2020-02-03
30STA0304 2021-02-22
30STA0619 2019-12-15
30STB2114 2019-02-03
[x] Data artifacts <- many probably due to brdf correction..
32SLD1007 2018-08-05
33KTA0515 2018-01-13
34HGJ2107 2021-03-25 & 2021-03-30
34JEL1409 2020-02-27
34JHT2020 2018-04-08
34LBH1705 2018-08-25
35JRN1209 2020-01-29
35MQV1400 2019-02-07, 12, 17, 22 & 27
[x] BRDF correction: The metadata does not properly cover the whole tile (if I take e.g. the upper left 512x512px of the tile, the metadata is not correctly interpolated). This is especially a problem in combination with the best orbit filter, since it might filter out dates which would actually be better in terms of the BRDF problem.... <- possible solutions: 1) if both brdf and orbit are used, then select best orbit by first getting brdf metadata and then computing % of non-nan pixels in there. 2) instead could also figure out what is the "brdf margin" and decrease the polygons reported in metadata by that margin 3) fix the brdf issue...
[x] Best orbit filter: There might be a consistency issue when using together with the monthly intervals... <- possible solutions: 1) adjust monthly intervals, such that the first and the second interval span > 1 month each, and in between are 1-month intervals, this should work... 2) put the best orbit calculation out of the monthly interval thing, e.g. by always giving the full interval also to the provider and computing the best orbit over that full interval at each monthly iteration
[ ] Value error.. Variables {'id'} are coordinates in some datasets but not others.... skipping /Net/Groups/BGI/scratch/DeepCube/UC1/minicube_cloudmasks_v03/35HKD0125.nc <- happens in sentinel 2 in the concat of the cloud masks.
[ ] Runtime error.. sequence argument must have length equal to input rank... skipping /Net/Groups/BGI/scratch/DeepCube/UC1/minicubes_v03/35MNM0003.nc <- this happens here: stack["s1_vv"] = stack.where(valid, 0).s1_vv.groupby("time").apply(FILTERS[self.speckle_filter_kwargs["type"]], size=self.speckle_filter_kwargs["size"])
[ ] pystac_client.exceptions.APIError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))for File "/Net/Groups/BGI/people/vbenson/EarthNet/earthnet-minicuber/earthnet_minicuber/provider/alos.py", line 33, in load_data items_dem = pc.sign(search)
[ ] RuntimeError: Error reading Window(col_off=0, row_off=0, width=189, height=191) from 's3://deafrica-sentinel-1/s1_rtc/N00E029/2019/04/02/01D4A6/s1_rtc_01D4A6_N00E02 9_2019_04_02_VH.tif': RasterioIOError('Read or write failed. IReadBlock failed at X offset 0, Y offset 0: /vsis3/deafrica-sentinel-1/s1_rtc/N00E029/2019/04/02/01D4A 6/s1_rtc_01D4A6_N00E029_2019_04_02_VH.tif, band 1: IReadBlock failed at X offset 4, Y offset 3: TIFFReadEncodedTile() failed.')
[ ] File /Net/Groups/BGI/work_1/scratch/DeepCube/en22/train/Lesotho/35JNG3013_local.nc already exist. Use option --overwrite to proceed anyway. Traceback (most recent call last): File "/Net/Groups/BGI/people/vbenson/EarthNet/earthnet-minicuber/earthnet_minicuber/minicuber.py", line 306, in save_minicube_mp cls.save_minicube(**pars) File "/Net/Groups/BGI/people/vbenson/EarthNet/earthnet-minicuber/earthnet_minicuber/minicuber.py", line 285, in save_minicube minicube = cls.load_minicube(specs, verbose = verbose, compute = True) File "/Net/Groups/BGI/people/vbenson/EarthNet/earthnet-minicuber/earthnet_minicuber/minicuber.py", line 197, in load_minicube product_cube = provider.load_data(self.padded_bbox, time_interval, full_time_interval = self.full_time_interval) File "/Net/Groups/BGI/people/vbenson/EarthNet/earthnet-minicuber/earthnet_minicuber/provider/s2/sentinel2.py", line 232, in load_data stack = computeCloudMask(aoi, stack, stack.time.values.astype('datetime64[Y]').astype(int)[0] + 1970) File "/Net/Groups/BGI/people/vbenson/EarthNet/earthnet-minicuber/earthnet_minicuber/provider/s2/sen2flux.py", line 243, in computeCloudMask CLOUD_MASK = CLOUD_MASK_xarray.CLOUD_MASK.interp( File "/Net/Groups/BGI/people/vbenson/miniconda3/envs/dea/lib/python3.9/site-packages/xarray/core/common.py", line 239, in __getattr__ raise AttributeError( AttributeError: 'Dataset' object has no attribute 'CLOUD_MASK' Unknown exception.. 'Dataset' object has no attribute 'CLOUD_MASK'... skipping /Net/Groups/BGI/work_1/scratch/DeepCube/en22/train/Congo/33NVA5550_local.nc
[ ] Traceback (most recent call last): File "/Net/Groups/BGI/people/vbenson/EarthNet/earthnet-minicuber/earthnet_minicuber/minicuber.py", line 306, in save_minicube_mp cls.save_minicube(**pars) File "/Net/Groups/BGI/people/vbenson/EarthNet/earthnet-minicuber/earthnet_minicuber/minicuber.py", line 285, in save_minicube minicube = cls.load_minicube(specs, verbose = verbose, compute = True) File "/Net/Groups/BGI/people/vbenson/EarthNet/earthnet-minicuber/earthnet_minicuber/minicuber.py", line 197, in load_minicube product_cube = provider.load_data(self.padded_bbox, time_interval, full_time_interval = self.full_time_interval) File "/Net/Groups/BGI/people/vbenson/EarthNet/earthnet-minicuber/earthnet_minicuber/provider/s2/sentinel2.py", line 232, in load_data stack = computeCloudMask(aoi, stack, stack.time.values.astype('datetime64[Y]').astype(int)[0] + 1970) IndexError: index 0 is out of bounds for axis 0 with size 0 Index error.. index 0 is out of bounds for axis 0 with size 0... skipping /Net/Groups/BGI/work_1/scratch/DeepCube/en22/test/Morocco/30SXB3654_local2.nc /Net/Groups/BGI/work_1/scratch/DeepCube/en22/test/Morocco/30SXB3654_local2.nc took 10.63 seconds.
[ ] when using many processes, it might happen that an OSError due to too many open files occurs. This is because there is a user limit of maximum number of open files (ulimit -n). Need to figure out workaround. Most importantly this happens when GEE tries to authenticate <- so maybe global lock for this works. It also happens if trying to read ERA5 from zarr <- maybe good idea to not read monthly, but do this for full timeseries.
[ ] Haze is still a problem:
29PLQ0212 2020-06-17
29RNQ1519 2021-07-10
30SXB2516 2020-03-06
[ ] Shadow fails
29RNQ1824 2018-11-13
30SUB1604 2020-12-29
[ ] Missmatch (?)
30SUB0618 2018-12-30
[ ] Bug with saving minicubes: apparently sometimes compression is not working? e.g. 29RNQ2517
[ ] EarthNet2021x generation generated some minicubes where merging of DEMs & Sen2 didn't work.. most likely happening in splitting into local & aws generation..
[ ] Soilgrids provider sometimes grabs locations too far inside the grid, so when regridding in minicuber, it puts NaN where it has to extrapolate
Found bugs:
Value error.. Variables {'id'} are coordinates in some datasets but not others.... skipping /Net/Groups/BGI/scratch/DeepCube/UC1/minicube_cloudmasks_v03/35HKD0125.nc<- happens in sentinel 2 in the concat of the cloud masks.Runtime error.. sequence argument must have length equal to input rank... skipping /Net/Groups/BGI/scratch/DeepCube/UC1/minicubes_v03/35MNM0003.nc<- this happens here:stack["s1_vv"] = stack.where(valid, 0).s1_vv.groupby("time").apply(FILTERS[self.speckle_filter_kwargs["type"]], size=self.speckle_filter_kwargs["size"])pystac_client.exceptions.APIError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))forFile "/Net/Groups/BGI/people/vbenson/EarthNet/earthnet-minicuber/earthnet_minicuber/provider/alos.py", line 33, in load_data items_dem = pc.sign(search)RuntimeError: Error reading Window(col_off=0, row_off=0, width=189, height=191) from 's3://deafrica-sentinel-1/s1_rtc/N00E029/2019/04/02/01D4A6/s1_rtc_01D4A6_N00E02 9_2019_04_02_VH.tif': RasterioIOError('Read or write failed. IReadBlock failed at X offset 0, Y offset 0: /vsis3/deafrica-sentinel-1/s1_rtc/N00E029/2019/04/02/01D4A 6/s1_rtc_01D4A6_N00E029_2019_04_02_VH.tif, band 1: IReadBlock failed at X offset 4, Y offset 3: TIFFReadEncodedTile() failed.')File /Net/Groups/BGI/work_1/scratch/DeepCube/en22/train/Lesotho/35JNG3013_local.nc already exist. Use option --overwrite to proceed anyway. Traceback (most recent call last): File "/Net/Groups/BGI/people/vbenson/EarthNet/earthnet-minicuber/earthnet_minicuber/minicuber.py", line 306, in save_minicube_mp cls.save_minicube(**pars) File "/Net/Groups/BGI/people/vbenson/EarthNet/earthnet-minicuber/earthnet_minicuber/minicuber.py", line 285, in save_minicube minicube = cls.load_minicube(specs, verbose = verbose, compute = True) File "/Net/Groups/BGI/people/vbenson/EarthNet/earthnet-minicuber/earthnet_minicuber/minicuber.py", line 197, in load_minicube product_cube = provider.load_data(self.padded_bbox, time_interval, full_time_interval = self.full_time_interval) File "/Net/Groups/BGI/people/vbenson/EarthNet/earthnet-minicuber/earthnet_minicuber/provider/s2/sentinel2.py", line 232, in load_data stack = computeCloudMask(aoi, stack, stack.time.values.astype('datetime64[Y]').astype(int)[0] + 1970) File "/Net/Groups/BGI/people/vbenson/EarthNet/earthnet-minicuber/earthnet_minicuber/provider/s2/sen2flux.py", line 243, in computeCloudMask CLOUD_MASK = CLOUD_MASK_xarray.CLOUD_MASK.interp( File "/Net/Groups/BGI/people/vbenson/miniconda3/envs/dea/lib/python3.9/site-packages/xarray/core/common.py", line 239, in __getattr__ raise AttributeError( AttributeError: 'Dataset' object has no attribute 'CLOUD_MASK' Unknown exception.. 'Dataset' object has no attribute 'CLOUD_MASK'... skipping /Net/Groups/BGI/work_1/scratch/DeepCube/en22/train/Congo/33NVA5550_local.ncTraceback (most recent call last): File "/Net/Groups/BGI/people/vbenson/EarthNet/earthnet-minicuber/earthnet_minicuber/minicuber.py", line 306, in save_minicube_mp cls.save_minicube(**pars) File "/Net/Groups/BGI/people/vbenson/EarthNet/earthnet-minicuber/earthnet_minicuber/minicuber.py", line 285, in save_minicube minicube = cls.load_minicube(specs, verbose = verbose, compute = True) File "/Net/Groups/BGI/people/vbenson/EarthNet/earthnet-minicuber/earthnet_minicuber/minicuber.py", line 197, in load_minicube product_cube = provider.load_data(self.padded_bbox, time_interval, full_time_interval = self.full_time_interval) File "/Net/Groups/BGI/people/vbenson/EarthNet/earthnet-minicuber/earthnet_minicuber/provider/s2/sentinel2.py", line 232, in load_data stack = computeCloudMask(aoi, stack, stack.time.values.astype('datetime64[Y]').astype(int)[0] + 1970) IndexError: index 0 is out of bounds for axis 0 with size 0 Index error.. index 0 is out of bounds for axis 0 with size 0... skipping /Net/Groups/BGI/work_1/scratch/DeepCube/en22/test/Morocco/30SXB3654_local2.nc /Net/Groups/BGI/work_1/scratch/DeepCube/en22/test/Morocco/30SXB3654_local2.nc took 10.63 seconds.29RNQ2517