fix(plugins): auth only when needed during http download

jlahovnik commented 1 month ago

So far if the download of one asset was failing, this was logged but no error was raised which hid problems during the download and could lead to incomplete or even empty results while the response was still 200. This is changed now: → in HTTPDownload if a problem occurs during an asset download, a Download error is raised and the whole product download is interrupted.
After this change it became visible that for planetary_computer some assets where not downloaded because, for the same product, there are assets where authentication is required for the download and assets where authentication should not be added. → In HTTPDownload, the asset url is now compared to matching_url from the config to check if authentication should be added to the request.

github-actions[bot] commented 1 month ago

Test Results

4 files ±0 4 suites ±0 5m 52s ⏱️ ±0s 573 tests +2 570 ✅ +2 3 💤 ±0 0 ❌ ±0 2 292 runs +8 2 196 ✅ +8 96 💤 ±0 0 ❌ ±0

Results for commit 3845bc2b. ± Comparison against base commit e20c12b4.

:recycle: This comment has been updated with latest results.

github-actions[bot] commented 1 month ago

Code Coverage (Ubuntu)

``` Filename Stmts Miss Cover Missing ----------------------------------------- ------- ------ ------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- __init__.py 8 0 100.00% cli.py 314 59 81.21% 657-716, 818-869, 873 config.py 429 27 93.71% 83-85, 94, 102, 106-108, 179, 190, 695-697, 810-813, 856-857, 866-867, 972, 1035-1040, 1042 crunch.py 5 5 0.00% 20-24 api/__init__.py 0 0 100.00% api/core.py 767 69 91.00% 374, 664, 708-711, 749, 793, 827, 872-877, 903, 994, 1062, 1200, 1285-1297, 1333, 1335, 1363, 1367-1378, 1391-1397, 1480-1483, 1516-1536, 1588, 1605-1609, 1621-1624, 1980, 2004-2010, 2261, 2265-2269, 2278-2280, 2312 api/search_result.py 58 4 93.10% 92, 101, 108, 122 api/product/__init__.py 6 0 100.00% api/product/_assets.py 48 5 89.58% 75, 147, 155, 158-162 api/product/_product.py 187 20 89.30% 70-72, 237-238, 313, 342, 399, 413-416, 429, 453-456, 499-505 api/product/metadata_mapping.py 680 68 90.00% 130-132, 227, 259-260, 306-307, 317-329, 331, 342, 348-360, 405-406, 443, 464-467, 490, 498-499, 575-576, 600-601, 607-610, 625-626, 775, 821, 992-997, 1124, 1138-1158, 1178, 1183, 1312, 1326, 1397, 1449, 1489-1493, 1508 api/product/drivers/__init__.py 6 0 100.00% api/product/drivers/base.py 6 1 83.33% 38 plugins/__init__.py 0 0 100.00% plugins/base.py 21 2 90.48% 48, 55 plugins/manager.py 172 15 91.28% 116-121, 171, 209, 231, 235, 259, 281, 399-402, 414-415 plugins/apis/__init__.py 0 0 100.00% plugins/apis/base.py 4 0 100.00% plugins/apis/ecmwf.py 91 8 91.21% 159-161, 208-209, 235-237 plugins/apis/usgs.py 180 31 82.78% 155, 257, 291, 326-328, 333, 359-360, 365, 395-402, 413-418, 440-446, 448-454, 477 plugins/authentication/__init__.py 6 1 83.33% 31 plugins/authentication/aws_auth.py 19 0 100.00% plugins/authentication/base.py 17 2 88.24% 43, 56 plugins/authentication/generic.py 14 2 85.71% 51, 61 plugins/authentication/header.py 19 0 100.00% plugins/authentication/keycloak.py 46 7 84.78% 151-154, 175-180 plugins/authentication/oauth.py 13 7 46.15% 42-44, 48-51 plugins/authentication/openid_connect.py 206 27 86.89% 80-81, 93-111, 149, 155-183, 191, 323-326, 352 plugins/authentication/qsauth.py 34 1 97.06% 91 plugins/authentication/sas_auth.py 47 1 97.87% 76 plugins/authentication/token.py 92 16 82.61% 114, 143, 145, 176-189, 245-249 plugins/authentication/token_exchange.py 36 20 44.44% 72-78, 90-120 plugins/crunch/__init__.py 0 0 100.00% plugins/crunch/base.py 10 1 90.00% 43 plugins/crunch/filter_date.py 59 14 76.27% 52-57, 69, 78, 87, 90, 102-104, 113-115, 122 plugins/crunch/filter_latest_intersect.py 47 8 82.98% 52-53, 69, 78-81, 83, 90-93 plugins/crunch/filter_latest_tpl_name.py 31 1 96.77% 83 plugins/crunch/filter_overlap.py 66 18 72.73% 28-30, 66-69, 76-79, 85, 93, 104-120 plugins/crunch/filter_property.py 30 7 76.67% 55-60, 63-64, 80-84 plugins/download/__init__.py 0 0 100.00% plugins/download/aws.py 491 163 66.80% 282, 295, 362-365, 379-383, 425-427, 431, 463-464, 470-474, 503, 535, 539, 546, 576-584, 588, 620-628, 639-641, 672-746, 764-824, 835-840, 852-865, 890, 905-907, 910, 920-928, 936-949, 959-990, 997-1009, 1047, 1073, 1118-1120, 1340 plugins/download/base.py 253 51 79.84% 136, 164, 296-297, 314-320, 351-355, 361-362, 404, 407-421, 433, 437, 501-505, 535-536, 544-561, 568-576, 578-582, 625, 647, 669, 677 plugins/download/creodias_s3.py 17 9 47.06% 53-67 plugins/download/http.py 537 108 79.89% 235, 331-334, 336-337, 344-349, 367-382, 399-401, 413, 461, 468-474, 492, 506, 520, 528-530, 546-551, 562, 580, 623-626, 648, 688, 733, 747-753, 782-846, 864, 894-903, 925-926, 955-960, 966, 969, 986, 1003-1004, 1034-1035, 1042, 1104, 1119, 1178-1179, 1185, 1195, 1231, 1267, 1287, 1322-1324 plugins/download/s3rest.py 116 24 79.31% 118, 154, 161, 196, 223-230, 233-235, 239, 250-256, 264-265, 268-272, 295, 316-319 plugins/search/__init__.py 22 0 100.00% plugins/search/base.py 132 13 90.15% 100, 104, 115, 274, 294, 350-351, 371, 374-382 plugins/search/build_search_result.py 188 24 87.23% 100, 141-142, 148, 159, 292-295, 324, 359-376, 425, 454, 457, 467, 484, 512, 514 plugins/search/cop_marine.py 240 47 80.42% 56, 64-66, 76-77, 82, 87-88, 104, 106, 109, 175-176, 218, 224, 228, 232, 245, 256-257, 265, 293, 297, 312, 316, 320, 324, 328-332, 338-341, 344-358, 375, 424-428, 433, 445 plugins/search/creodias_s3.py 55 3 94.55% 58, 76, 110 plugins/search/csw.py 105 81 22.86% 98-99, 103-104, 112-160, 166-179, 187-219, 237-278 plugins/search/data_request_search.py 202 69 65.84% 188-191, 207, 218, 222-223, 234, 239, 244, 251, 264-267, 321-322, 326, 336-342, 347, 373-376, 384-395, 412, 414, 421-424, 426-427, 445-449, 482, 492, 503, 516, 522-537, 542 plugins/search/qssearch.py 774 103 86.69% 458, 516, 530, 534-540, 568-572, 681-693, 737, 753, 763, 782-797, 834-837, 908-909, 957, 976, 983, 995, 1052, 1073, 1076-1077, 1086-1087, 1096-1097, 1106-1107, 1134, 1205-1210, 1214-1223, 1257, 1279, 1356, 1446, 1533, 1536-1537, 1619-1623, 1685, 1688, 1694-1695, 1716, 1743-1755, 1762, 1794-1796, 1806-1812, 1842, 1865, 1870-1871, 1886, 1902, 1987-1990, 1995-1998, 2007, 2027-2028, 2037-2041, 2047 plugins/search/static_stac_search.py 75 13 82.67% 98-124, 140, 153 rest/__init__.py 4 2 50.00% 21-22 rest/cache.py 33 7 78.79% 35-37, 53-55, 59, 68 rest/config.py 25 0 100.00% rest/constants.py 6 0 100.00% rest/core.py 251 63 74.90% 216-217, 273, 281, 299-316, 331-369, 460, 502-533, 680, 687-735 rest/errors.py 69 5 92.75% 106, 116, 127, 143-144 rest/server.py 188 24 87.23% 89, 112-114, 277-282, 310, 495-497, 514-519, 548, 550, 554-555, 559-560 rest/stac.py 319 63 80.25% 306, 328, 380-383, 410-437, 468-470, 493, 525-526, 608-648, 670-686, 778-782, 789, 843-844, 905, 995-997 rest/types/__init__.py 0 0 100.00% rest/types/collections_search.py 13 13 0.00% 18-44 rest/types/eodag_search.py 176 5 97.16% 225-229, 282, 285, 353 rest/types/queryables.py 55 1 98.18% 163 rest/types/stac_search.py 125 7 94.40% 138, 184, 199-201, 209, 213 rest/utils/__init__.py 93 12 87.10% 108-109, 128-130, 182, 192-206 rest/utils/cql_evaluate.py 48 5 89.58% 69, 76, 90, 97, 105 rest/utils/rfc3339.py 23 3 86.96% 48, 60, 62 types/__init__.py 114 14 87.72% 65, 82, 141-144, 211, 225-234, 244, 265, 278 types/bbox.py 43 19 55.81% 46-61, 72-74, 85-87, 99-101, 113-115, 123 types/download_args.py 9 0 100.00% types/queryables.py 80 0 100.00% types/search_args.py 70 18 74.29% 60-64, 71-88, 103 types/whoosh.py 15 0 100.00% utils/__init__.py 502 36 92.83% 85, 193-194, 203-230, 233, 248, 328-332, 405-409, 428-430, 509, 524, 564-565, 929-932, 940-941, 979-980, 1151 utils/constraints.py 119 38 68.07% 62, 89-98, 139, 144, 148, 159, 182-184, 194, 208-224, 233-244 utils/exceptions.py 41 0 100.00% utils/import_system.py 28 19 32.14% 64-78, 89-99 utils/logging.py 28 1 96.43% 41 utils/notebook.py 44 23 47.73% 25-29, 36-41, 58-62, 72-78, 83-87 utils/repr.py 30 8 73.33% 36, 38, 42, 76, 94-101 utils/requests.py 55 11 80.00% 64, 86, 88, 90, 92, 94, 110, 118-120, 128 utils/rest.py 36 1 97.22% 55 utils/stac_reader.py 111 45 59.46% 56-57, 63-85, 95-97, 101, 137, 153-156, 203-212, 222-252 TOTAL 9734 1598 83.58% ```

Diff against develop

Filename                    Stmts    Miss  Cover
------------------------  -------  ------  -------
plugins/download/http.py       +3     -12  +2.36%
TOTAL                          +3     -12  +0.13%

Results for commit: 3845bc2ba550ad2fac609343508e7098416ca96e

Minimum allowed coverage is 70%

:recycle: This comment has been updated with latest results

github-actions[bot] commented 1 month ago

Code Coverage (Windows)

``` Filename Stmts Miss Cover Missing ----------------------------------------- ------- ------ ------- --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- __init__.py 8 0 100.00% cli.py 314 59 81.21% 657-716, 818-869, 873 config.py 429 28 93.47% 83-85, 94, 102, 106-108, 179, 190, 695-697, 810-813, 856-857, 866-867, 972, 1003, 1035-1040, 1042 crunch.py 5 5 0.00% 20-24 api/__init__.py 0 0 100.00% api/core.py 767 69 91.00% 374, 664, 708-711, 749, 793, 827, 872-877, 903, 994, 1062, 1200, 1285-1297, 1333, 1335, 1363, 1367-1378, 1391-1397, 1480-1483, 1516-1536, 1588, 1605-1609, 1621-1624, 1980, 2004-2010, 2261, 2265-2269, 2278-2280, 2312 api/search_result.py 58 4 93.10% 92, 101, 108, 122 api/product/__init__.py 6 0 100.00% api/product/_assets.py 48 5 89.58% 75, 147, 155, 158-162 api/product/_product.py 187 20 89.30% 70-72, 237-238, 313, 342, 399, 413-416, 429, 453-456, 499-505 api/product/metadata_mapping.py 680 68 90.00% 130-132, 227, 259-260, 306-307, 317-329, 331, 342, 348-360, 405-406, 443, 464-467, 490, 498-499, 575-576, 600-601, 607-610, 625-626, 775, 821, 992-997, 1124, 1138-1158, 1178, 1183, 1312, 1326, 1397, 1449, 1489-1493, 1508 api/product/drivers/__init__.py 6 0 100.00% api/product/drivers/base.py 6 1 83.33% 38 plugins/__init__.py 0 0 100.00% plugins/base.py 21 3 85.71% 48, 55, 68 plugins/manager.py 172 15 91.28% 116-121, 171, 209, 231, 235, 259, 281, 399-402, 414-415 plugins/apis/__init__.py 0 0 100.00% plugins/apis/base.py 4 0 100.00% plugins/apis/ecmwf.py 91 8 91.21% 159-161, 208-209, 235-237 plugins/apis/usgs.py 180 31 82.78% 155, 257, 291, 326-328, 333, 359-360, 365, 395-402, 413-418, 440-446, 448-454, 477 plugins/authentication/__init__.py 6 1 83.33% 31 plugins/authentication/aws_auth.py 19 0 100.00% plugins/authentication/base.py 17 2 88.24% 43, 56 plugins/authentication/generic.py 14 2 85.71% 51, 61 plugins/authentication/header.py 19 0 100.00% plugins/authentication/keycloak.py 46 7 84.78% 151-154, 175-180 plugins/authentication/oauth.py 13 7 46.15% 42-44, 48-51 plugins/authentication/openid_connect.py 206 27 86.89% 80-81, 93-111, 149, 155-183, 191, 323-326, 352 plugins/authentication/qsauth.py 34 1 97.06% 91 plugins/authentication/sas_auth.py 47 1 97.87% 76 plugins/authentication/token.py 92 16 82.61% 114, 143, 145, 176-189, 245-249 plugins/authentication/token_exchange.py 36 20 44.44% 72-78, 90-120 plugins/crunch/__init__.py 0 0 100.00% plugins/crunch/base.py 10 1 90.00% 43 plugins/crunch/filter_date.py 59 14 76.27% 52-57, 69, 78, 87, 90, 102-104, 113-115, 122 plugins/crunch/filter_latest_intersect.py 47 33 29.79% 49-54, 67-112 plugins/crunch/filter_latest_tpl_name.py 31 1 96.77% 83 plugins/crunch/filter_overlap.py 66 18 72.73% 28-30, 66-69, 76-79, 85, 93, 104-120 plugins/crunch/filter_property.py 30 7 76.67% 55-60, 63-64, 80-84 plugins/download/__init__.py 0 0 100.00% plugins/download/aws.py 491 163 66.80% 282, 295, 362-365, 379-383, 425-427, 431, 463-464, 470-474, 503, 535, 539, 546, 576-584, 588, 620-628, 639-641, 672-746, 764-824, 835-840, 852-865, 890, 905-907, 910, 920-928, 936-949, 959-990, 997-1009, 1047, 1073, 1118-1120, 1340 plugins/download/base.py 253 53 79.05% 136, 164, 231-233, 296-297, 314-320, 351-355, 361-362, 404, 407-421, 433, 437, 501-505, 535-536, 544-561, 568-576, 578-582, 625, 647, 669, 677 plugins/download/creodias_s3.py 17 9 47.06% 53-67 plugins/download/http.py 537 109 79.70% 235, 331-334, 336-337, 344-349, 367-382, 399-401, 413, 461, 468-474, 492, 506, 520, 528-530, 546-551, 562, 580, 623-626, 648, 688, 733, 747-753, 782-846, 864, 894-903, 925-926, 955-960, 966, 969, 986, 1003-1004, 1017, 1034-1035, 1042, 1104, 1119, 1178-1179, 1185, 1195, 1231, 1267, 1287, 1322-1324 plugins/download/s3rest.py 116 24 79.31% 118, 154, 161, 196, 223-230, 233-235, 239, 250-256, 264-265, 268-272, 295, 316-319 plugins/search/__init__.py 22 0 100.00% plugins/search/base.py 132 14 89.39% 100, 104, 115, 274, 294, 350-351, 371, 374-382, 384 plugins/search/build_search_result.py 188 31 83.51% 100, 141-142, 148, 159, 292-295, 324, 359-376, 425, 454, 457, 467, 484, 504-519 plugins/search/cop_marine.py 240 47 80.42% 56, 64-66, 76-77, 82, 87-88, 104, 106, 109, 175-176, 218, 224, 228, 232, 245, 256-257, 265, 293, 297, 312, 316, 320, 324, 328-332, 338-341, 344-358, 375, 424-428, 433, 445 plugins/search/creodias_s3.py 55 3 94.55% 58, 76, 110 plugins/search/csw.py 105 81 22.86% 98-99, 103-104, 112-160, 166-179, 187-219, 237-278 plugins/search/data_request_search.py 202 69 65.84% 188-191, 207, 218, 222-223, 234, 239, 244, 251, 264-267, 321-322, 326, 336-342, 347, 373-376, 384-395, 412, 414, 421-424, 426-427, 445-449, 482, 492, 503, 516, 522-537, 542 plugins/search/qssearch.py 774 135 82.56% 458, 516, 530, 534-540, 568-572, 681-693, 737, 740, 753, 763, 782-797, 834-837, 908-909, 957, 976, 983, 995, 1052, 1073, 1076-1077, 1086-1087, 1096-1097, 1106-1107, 1134, 1205-1210, 1214-1223, 1257, 1279, 1356, 1446, 1533, 1536-1537, 1619-1623, 1685, 1688, 1694-1695, 1716, 1743-1755, 1762, 1794-1796, 1806-1812, 1842, 1865, 1870-1871, 1886, 1902, 1970-2076 plugins/search/static_stac_search.py 75 13 82.67% 98-124, 140, 153 rest/__init__.py 4 2 50.00% 21-22 rest/cache.py 33 22 33.33% 35-37, 44-70 rest/config.py 25 1 96.00% 35 rest/constants.py 6 0 100.00% rest/core.py 251 139 44.62% 162, 164, 166, 169-170, 184-194, 206-207, 209-210, 216-217, 220, 223, 264-318, 331-369, 400-434, 449-465, 481-490, 502-533, 550, 592-641, 680, 687-735 rest/errors.py 69 49 28.99% 60, 65-100, 105-108, 115-118, 126-147, 155-160, 175-181 rest/server.py 188 188 0.00% 18-573 rest/stac.py 319 68 78.68% 240, 306, 328, 380-383, 410-437, 468-470, 493, 525-526, 608-648, 670-686, 713, 778-782, 789, 843-844, 850, 905, 943, 976, 995-997 rest/types/__init__.py 0 0 100.00% rest/types/collections_search.py 13 13 0.00% 18-44 rest/types/eodag_search.py 176 16 90.91% 225-229, 262-264, 282, 285, 291, 295, 353, 370-380 rest/types/queryables.py 55 13 76.36% 50-51, 58-59, 66-67, 93-98, 107-108, 163 rest/types/stac_search.py 125 11 91.20% 136-138, 184, 199-201, 209, 213, 261, 264 rest/utils/__init__.py 93 30 67.74% 79-85, 105, 108-109, 128-130, 143, 150, 175-183, 190-211 rest/utils/cql_evaluate.py 48 5 89.58% 69, 76, 90, 97, 105 rest/utils/rfc3339.py 23 5 78.26% 43-44, 48, 60, 62 types/__init__.py 114 39 65.79% 65, 78-82, 93-105, 132-134, 141-144, 184, 211, 221-237, 242, 244, 265, 270, 278, 288 types/bbox.py 43 19 55.81% 46-61, 72-74, 85-87, 99-101, 113-115, 123 types/download_args.py 9 0 100.00% types/queryables.py 80 0 100.00% types/search_args.py 70 18 74.29% 60-64, 71-88, 103 types/whoosh.py 15 0 100.00% utils/__init__.py 502 36 92.83% 85, 193-194, 203-230, 233, 248, 328-332, 405-409, 428-430, 509, 524, 564-565, 929-932, 940-941, 979-980, 1151 utils/constraints.py 119 38 68.07% 62, 89-98, 139, 144, 148, 159, 182-184, 194, 208-224, 233-244 utils/exceptions.py 41 0 100.00% utils/import_system.py 28 19 32.14% 64-78, 89-99 utils/logging.py 28 1 96.43% 41 utils/notebook.py 44 23 47.73% 25-29, 36-41, 58-62, 72-78, 83-87 utils/repr.py 30 8 73.33% 36, 38, 42, 76, 94-101 utils/requests.py 55 11 80.00% 64, 86, 88, 90, 92, 94, 110, 118-120, 128 utils/rest.py 36 1 97.22% 55 utils/stac_reader.py 111 45 59.46% 56-57, 63-85, 95-97, 101, 137, 153-156, 203-212, 222-252 TOTAL 9734 2045 78.99% ```

Diff against develop

Filename                    Stmts    Miss  Cover
------------------------  -------  ------  -------
plugins/download/http.py       +3     -12  +2.36%
TOTAL                          +3     -12  +0.13%

Results for commit: 3845bc2ba550ad2fac609343508e7098416ca96e

Minimum allowed coverage is 70%

:recycle: This comment has been updated with latest results

CS-SI / eodag