search

SpatialDays / catapult-workfinder

Tooling to find work to do for common sensing pipelines.
Other
0 stars 0 forks source link

limit parameter unsafe #2

Closed ivica3730k closed 1 year ago

ivica3730k commented 2 years ago

If the limit is not set it processes the whole collection, but if you specify any other limit, it only places one order.


(env) ivica@icepad-2:~/Coding/new-ard/catapult-workfinder$ python3 -m workfinder.workfinder LANDSAT8 --limit 100
2022-11-09 13:57:09,453 DEBUG shapely.geos geos.py:73 -- Found GEOS DLL: <CDLL '/home/ivica/Coding/new-ard/env/lib/python3.8/site-packages/shapely/.libs/libgeos_c-a68605fd.so.1.13.1', handle 31a0c50 at 0x7f032945b9a0>, using it.
2022-11-09 13:57:09,461 DEBUG shapely.geos geos.py:38 -- Trying `CDLL(libc.so.6)`
2022-11-09 13:57:09,462 DEBUG shapely.geos geos.py:55 -- Library path: 'libc.so.6'
2022-11-09 13:57:09,462 DEBUG shapely.geos geos.py:56 -- DLL: <CDLL 'libc.so.6', handle 7f0372f12000 at 0x7f032945b9d0>
Limit is:  100
2022-11-09 13:57:09,501 INFO root workfinder.py:64 -- starting work search for LANDSAT8
2022-11-09 13:57:09,551 INFO root __init__.py:70 -- TM_WORLD_BORDERS.geojson already available
/home/ivica/Coding/new-ard/env/lib/python3.8/site-packages/pyproj/crs/crs.py:141: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable/gotchas.html#axis-order-changes-in-proj-6
  in_crs_string = _prepare_from_proj_string(in_crs_string)
/home/ivica/Coding/new-ard/env/lib/python3.8/site-packages/pyproj/crs/crs.py:141: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable/gotchas.html#axis-order-changes-in-proj-6
  in_crs_string = _prepare_from_proj_string(in_crs_string)
2022-11-09 13:57:10,920 INFO root __init__.py:70 -- WRS2_descending.geojson already available
2022-11-09 13:57:22,927 INFO root Landsat.py:106 -- downloading landsat8 metadata
2022-11-09 13:57:22,927 INFO root __init__.py:84 -- LANDSAT_OT_C2_L2.csv.gz already available
2022-11-09 13:57:58,792 INFO root Landsat.py:110 -- got metadata: 115372656
2022-11-09 13:57:58,792 INFO root Landsat.py:27 -- finding scenes
2022-11-09 13:57:59,327 INFO root Landsat.py:32 -- converting to required objects
2022-11-09 13:57:59,327 INFO root Landsat.py:33 -- Index(['Browse Link', 'Display ID', 'Ordering ID',
       'Landsat Product Identifier L2', 'Landsat Product Identifier L1',
       'Landsat Scene Identifier', 'Date Acquired', 'Collection Category',
       'Collection Number', 'WRS Path', 'WRS Row', 'Target WRS Path',
       'Target WRS Row', 'Nadir/Off Nadir', 'Roll Angle',
       'Date Product Generated L2', 'Date Product Generated L1', 'Start Time',
       'Stop Time', 'Station Identifier', 'Day/Night Indicator',
       'Land Cloud Cover', 'Scene Cloud Cover L1',
       'Ground Control Points Model', 'Ground Control Points Version',
       'Geometric RMSE Model', 'Geometric RMSE Model X',
       'Geometric RMSE Model Y', 'Processing Software Version',
       'Sun Elevation L0RA', 'Sun Azimuth L0RA', 'TIRS SSM Model',
       'Data Type L2', 'Sensor Identifier', 'Satellite',
       'Product Map Projection L1', 'UTM Zone', 'Datum', 'Ellipsoid',
       'Scene Center Latitude', 'Scene Center Longitude',
       'Corner Upper Left Latitude', 'Corner Upper Left Longitude',
       'Corner Upper Right Latitude', 'Corner Upper Right Longitude',
       'Corner Lower Left Latitude', 'Corner Lower Left Longitude',
       'Corner Lower Right Latitude', 'Corner Lower Right Longitude'],
      dtype='object')
2022-11-09 13:57:59,328 INFO root Landsat.py:34 --                                             Browse Link  ... Corner Lower Right Longitude
1949  https://landsatlook.usgs.gov/gen-browse?size=r...  ...                   -138.81163

[1 rows x 49 columns]
/home/ivica/Coding/new-ard/env/lib/python3.8/site-packages/urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host 's3.eu-west-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
2022-11-09 13:58:01,497 INFO root __init__.py:103 -- got 0 files, for 'common_sensing/fiji/landsat_8/'
2022-11-09 13:58:01,498 INFO root __init__.py:112 -- found 0 entries
2022-11-09 13:58:01,499 INFO root BaseWorkFinder.py:55 -- Checking 29630 things against 0 already done things.
2022-11-09 13:58:01,512 INFO root BaseWorkFinder.py:63 -- found 29630 things to do.
2022-11-09 13:58:01,512 INFO root BaseWorkFinder.py:64 --                                 id                                       url
1949     LC08_L1GT_069011_20200909  LC08_L1GT_069011_20200909_20200919_02_T2
1950     LC08_L1TP_069012_20200909  LC08_L1TP_069012_20200909_20200919_02_T1
1951     LC08_L1TP_069013_20200909  LC08_L1TP_069013_20200909_20200919_02_T1
1952     LC08_L1TP_069014_20200909  LC08_L1TP_069014_20200909_20200919_02_T1
1953     LC08_L1GT_069015_20200909  LC08_L1GT_069015_20200909_20200919_02_T2
...                            ...                                       ...
2352221  LC08_L1TP_073090_20170727  LC08_L1TP_073090_20170727_20200903_02_T1
2352222  LC08_L1TP_073091_20170727  LC08_L1TP_073091_20170727_20200903_02_T1
2352223  LC08_L1GT_073240_20170727  LC08_L1GT_073240_20170727_20200903_02_T2
2352224  LC08_L1TP_073241_20170727  LC08_L1TP_073241_20170727_20200903_02_T1
2352225  LC08_L1TP_073242_20170727  LC08_L1TP_073242_20170727_20200903_02_T1

[29630 rows x 2 columns]
Work size is:  59260
2022-11-09 13:58:02,092 DEBUG root espa_api.py:16 -- 200 OK
2022-11-09 13:58:02,092 INFO root Landsat.py:96 -- {"olitirs8_collection_2_l1": {"products": ["l1", "aq_refl", "stats"], "inputs": ["LC08_L1GT_070248_20150514_20201016_02_T2", "LC08_L1TP_071015_20130515_20200913_02_T1", "LC08_L1GT_070069_20200730_20200923_02_T2", "LC08_L1TP_070066_20180725_20201016_02_T1", "LC08_L1GT_070021_20200730_20200908_02_T2", "LC08_L1TP_073085_20170406_20200904_02_T1", "LC08_L1TP_071242_20220727_20220802_02_T1", "LC08_L1TP_070021_20180911_20200830_02_T1", "LC08_L1TP_070063_20210903_20210910_02_T1", "LC08_L1TP_073070_20210401_20210409_02_T1", "LC08_L1TP_071069_20150622_20201016_02_T1", "LC08_L1TP_071011_20140603_20200911_02_T1", "LC08_L1GT_070010_20150514_20201015_02_T2", "LC08_L1TP_071071_20150622_20201016_02_T2", "LC08_L1TP_072075_20161208_20201016_02_T1", "LC08_L1GT_073240_20190530_20200830_02_T2", "LC08_L1TP_071048_20220406_20220412_02_T1", "LC08_L1TP_073089_20190311_20200829_02_T1", "LC08_L1TP_070071_20141221_20201016_02_T2", "LC08_L1GT_072110_20211120_20211130_02_T2", "LC08_L1TP_069019_20200707_20200912_02_T1", "LC08_L1TP_072070_20180824_20201016_02_T1", "LC08_L1GT_073042_20170201_20201016_02_T2", "LC08_L1GT_069072_20211030_20211109_02_T2", "LC08_L1GT_072109_20200102_20201016_02_T2", "LC08_L1GT_071008_20150505_20200909_02_T2", "LC08_L1GT_072043_20220718_20220725_02_T2", "LC08_L1GT_072043_20151222_20201016_02_T2", "LC08_L1TP_071241_20200705_20200913_02_T1", "LC08_L1TP_070016_20220602_20220609_02_T1", "LC08_L1TP_069063_20221017_20221031_02_T1", "LC08_L1TP_070074_20170722_20201016_02_T1", "LC08_L1GT_070112_20141221_20201016_02_T2", "LC08_L1TP_071016_20130324_20200913_02_T1", "LC08_L1GT_069108_20200214_20201016_02_T2", "LC08_L1TP_073017_20200617_20200823_02_T1", "LC08_L1GT_070017_20200308_20200822_02_T2", "LC08_L1TP_073241_20200516_20200820_02_T1", "LC08_L1TP_069069_20180819_20201016_02_T1", "LC08_L1TP_070074_20200104_20201016_02_T1", "LC08_L1TP_070019_20160921_20200906_02_T1", "LC08_L1TP_073019_20180527_20200901_02_T1", "LC08_L1TP_073241_20220725_20220802_02_T1", "LC08_L1TP_070071_20150530_20201016_02_T1", "LC08_L1GT_070018_20170823_20200903_02_T2", "LC08_L1TP_071063_20131123_20201016_02_T1", "LC08_L1GT_069009_20150507_20200909_02_T2", "LC08_L1TP_072072_20170618_20201016_02_T1", "LC08_L1TP_073017_20190615_20200828_02_T1", "LC08_L1GT_070070_20130524_20201016_02_T2", "LC08_L1TP_072019_20170720_20200903_02_T1", "LC08_L1GT_072006_20150528_20200909_02_T2", "LC08_L1GT_070070_20131218_20201016_02_T2", "LC08_L1TP_073015_20220927_20221004_02_T1", "LC08_L1GT_072121_20141219_20201016_02_T2", "LC08_L1TP_072073_20180301_20201016_02_T1", "LC08_L1GT_069114_20191212_20201022_02_T2", "LC08_L1GT_070073_20131116_20201016_02_T2", "LC08_L1TP_073013_20151010_20200908_02_T1", "LC08_L1GT_069021_20191110_20200825_02_T2", "LC08_L1TP_069074_20170528_20201016_02_T1", "LC08_L1TP_070016_20220906_20220914_02_T1", "LC08_L1TP_073021_20170929_20200903_02_T1", "LC08_L1GT_072043_20141203_20201016_02_T2", "LC08_L1TP_071011_20160726_20200906_02_T1", "LC08_L1TP_070012_20190914_20200826_02_T1", "LC08_L1GT_071086_20180918_20200830_02_T2", "LC08_L1TP_071089_20191007_20200825_02_T1", "LC08_L1TP_070013_20200324_20200822_02_T1", "LC08_L1TP_070076_20170503_20201016_02_T2", "LC08_L1GT_073022_20200414_20200822_02_T2", "LC08_L1TP_070063_20190930_20201016_02_T1", "LC08_L1GT_069021_20200402_20200822_02_T2", "LC08_L1GT_072086_20210528_20210608_02_T2", "LC08_L1GT_071072_20160523_20201016_02_T2", "LC08_L1GT_069110_20200129_20201016_02_T2", "LC08_L1TP_072019_20141016_20200911_02_T1", "LC08_L1GT_070080_20150122_20201016_02_T2", "LC08_L1GT_073060_20161231_20201016_02_T2", "LC08_L1GT_069044_20140808_20201016_02_T2", "LC08_L1GT_071070_20150505_20201016_02_T2", "LC08_L1TP_072089_20151104_20200908_02_T1", "LC08_L1GT_069006_20220713_20220722_02_T2", "LC08_L1TP_069075_20151217_20201016_02_T1", "LC08_L1TP_072241_20200728_20200908_02_T1", "LC08_L1TP_070043_20190117_20201016_02_T2", "LC08_L1GT_072061_20170720_20201016_02_T2", "LC08_L1TP_070043_20151122_20201016_02_T1", "LC08_L1GT_070019_20170807_20200903_02_T2", "LC08_L1TP_069066_20171019_20201016_02_T1", "LC08_L1TP_073091_20191224_20200824_02_T1", "LC08_L1GT_070066_20200714_20200923_02_T2", "LC08_L1TP_071087_20200212_20200823_02_T1", "LC08_L1TP_073019_20190818_20200827_02_T1", "LC08_L1GT_072108_20140930_20201016_02_T2", "LC08_L1GT_073015_20130630_20200912_02_T2", "LC08_L1GT_073112_20151213_20201016_02_T2", "LC08_L1TP_070020_20200714_20200912_02_T1", "LC08_L1TP_069014_20170528_20200903_02_T1", "LC08_L1GT_073113_20141226_20201016_02_T2"]}, "format": "gtiff", "resampling_method": "cc", "note": "CS_Fiji_regular", "projection": {"aea": {"standard_parallel_1": 29.5, "standard_parallel_2": 45.5, "central_meridian": -96.0, "latitude_of_origin": 23.0, "false_easting": 0, "false_northing": 0, "datum": "nad83"}}}
2022-11-09 13:58:05,341 DEBUG root espa_api.py:16 -- 201 CREATED
2022-11-09 13:58:05,341 INFO root Landsat.py:101 -- created order id espa-ivica.matic@spatialdays.com-11092022-075805-173
2022-11-09 13:58:05,344 INFO root workfinder.py:72 -- done work search for LANDSAT8
ivica3730k commented 2 years ago

pandas .size() function returns the number of cells, not the number of rows image