Various issue when trying to install onto ubuntu

permezel commented 1 year ago

Expected behaviour

Less errors.

Actual behaviour

dap@odc ~/proj/datacube-core % datacube system check 
Version:       1.8.8
Config files:  /home/dap/.datacube.conf
Host:          localhost:5432
Database:      datacube
User:          dap
Environment:   None
Index Driver:  default

Valid connection:   2022-12-10 23:24:51,360 510586 datacube.drivers.driver_cache WARNING Failed to resolve driver datacube.plugins.index::default
2022-12-10 23:24:51,360 510586 datacube.drivers.driver_cache WARNING Error was: VersionConflict(rasterio 1.2.10 (/home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages), Requirement.parse('rasterio>=1.3.2'))
2022-12-10 23:24:51,361 510586 datacube.drivers.driver_cache WARNING Failed to resolve driver datacube.plugins.index::memory
2022-12-10 23:24:51,361 510586 datacube.drivers.driver_cache WARNING Error was: VersionConflict(rasterio 1.2.10 (/home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages), Requirement.parse('rasterio>=1.3.2'))
2022-12-10 23:24:51,361 510586 datacube.drivers.driver_cache WARNING Failed to resolve driver datacube.plugins.index::null
2022-12-10 23:24:51,361 510586 datacube.drivers.driver_cache WARNING Error was: VersionConflict(rasterio 1.2.10 (/home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages), Requirement.parse('rasterio>=1.3.2'))
2022-12-10 23:24:51,361 510586 datacube.drivers.driver_cache WARNING Failed to resolve driver datacube.plugins.index::postgis
2022-12-10 23:24:51,361 510586 datacube.drivers.driver_cache WARNING Error was: VersionConflict(rasterio 1.2.10 (/home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages), Requirement.parse('rasterio>=1.3.2'))

Steps to reproduce the behaviour

Installed using https://datacube-core.readthedocs.io/en/latest/installation/setup/ubuntu.html Ran into lots of errors, but my setup script is now:

set -x
PYTHON=${1-3.8}
ENV=odc_${PYTHON}
conda env remove -n ${ENV} --yes
conda config --append channels conda-forge
conda update -n base --yes -c defaults conda
conda create --name ${ENV} --yes python=${PYTHON} datacube
conda install -n ${ENV} --yes pycodestyle
conda install -n ${ENV} --yes pylint
conda install -n ${ENV} --yes jupyter matplotlib scipy pytest-cov hypothesis
conda install -n ${ENV} --yes geoalchemy2 moto 
cat ~/.datacube_integration.conf
cd ~/proj/datacube-core
conda run -n ${ENV} ./check-code.sh integration_tests

The integration tests fail with many errors. So I decided to try to init the database following https://datacube-core.readthedocs.io/en/latest/installation/database/setup.html

I created a database:

postgres@odc:~$ createdb datacube

I created another config file:

dap@odc ~/proj/datacube-core % cat ~/.datacube.conf 
[datacube]
# One config file may contain multiple named sections providing multiple configuration environments.
# The section named "datacube" (or "default") is used if no environment is specified.

# index_driver is optional and defaults to "default" (the default Postgres index driver)
index_driver: default

# The remaining configuration entries are for the default Postgres index driver and
# may not apply to other index drivers.
db_database: datacube

# A blank host will use a local socket. Specify a hostname (such as localhost) to use TCP.
db_hostname: localhost

# Credentials are optional: you might have other Postgres authentication configured.
# The default username otherwise is the current user id.
db_username: dap
db_password: postgres4me

[test]
# A "test" environment that accesses a separate test database.
index_driver: default
db_database: datacube_test

[null]
# A "null" environment for working with no index.
index_driver: null

[local_memory]
# A local non-persistent in-memory index.
#   Compatible with the default index driver, but resides purely in memory with no persistent database.
#   Note that each new invocation will receive a new, empty index.
index_driver: memory
dap@odc ~/proj/datacube-core %

I then tried to init it:

dap@odc ~/proj/datacube-core % datacube -v system init
2022-12-10 23:34:01,711 512304 datacube INFO Running datacube command: /home/dap/anaconda3/envs/odc_3.8/bin/datacube -v system init
2022-12-10 23:34:01,795 512304 datacube.drivers.driver_cache WARNING Failed to resolve driver datacube.plugins.index::default
2022-12-10 23:34:01,795 512304 datacube.drivers.driver_cache WARNING Error was: VersionConflict(rasterio 1.2.10 (/home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages), Requirement.parse('rasterio>=1.3.2'))
2022-12-10 23:34:01,795 512304 datacube.drivers.driver_cache WARNING Failed to resolve driver datacube.plugins.index::memory
2022-12-10 23:34:01,795 512304 datacube.drivers.driver_cache WARNING Error was: VersionConflict(rasterio 1.2.10 (/home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages), Requirement.parse('rasterio>=1.3.2'))
2022-12-10 23:34:01,796 512304 datacube.drivers.driver_cache WARNING Failed to resolve driver datacube.plugins.index::null
2022-12-10 23:34:01,796 512304 datacube.drivers.driver_cache WARNING Error was: VersionConflict(rasterio 1.2.10 (/home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages), Requirement.parse('rasterio>=1.3.2'))
2022-12-10 23:34:01,796 512304 datacube.drivers.driver_cache WARNING Failed to resolve driver datacube.plugins.index::postgis
2022-12-10 23:34:01,796 512304 datacube.drivers.driver_cache WARNING Error was: VersionConflict(rasterio 1.2.10 (/home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages), Requirement.parse('rasterio>=1.3.2'))
Initialising database...
2022-12-10 23:34:01,808 512304 datacube.drivers.postgres._core INFO Ensuring user roles.
2022-12-10 23:34:01,811 512304 datacube.drivers.postgres._core INFO Adding role grants.
2022-12-10 23:34:01,813 512304 datacube.drivers.postgres._core INFO No schema updates required.
Updated.
Checking indexes/views.
2022-12-10 23:34:01,813 512304 datacube.drivers.postgres._api INFO Checking dynamic views/indexes. (rebuild views=True, indexes=False)
Done.
dap@odc ~/proj/datacube-core %

Due to the verbiage spewed as a result of this, I tried to check:

dap@odc ~/proj/datacube-core % datacube system check 
Version:       1.8.8
Config files:  /home/dap/.datacube.conf
Host:          localhost:5432
Database:      datacube
User:          dap
Environment:   None
Index Driver:  default

Valid connection:   2022-12-10 23:35:49,773 512681 datacube.drivers.driver_cache WARNING Failed to resolve driver datacube.plugins.index::default
2022-12-10 23:35:49,773 512681 datacube.drivers.driver_cache WARNING Error was: VersionConflict(rasterio 1.2.10 (/home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages), Requirement.parse('rasterio>=1.3.2'))
2022-12-10 23:35:49,774 512681 datacube.drivers.driver_cache WARNING Failed to resolve driver datacube.plugins.index::memory
2022-12-10 23:35:49,774 512681 datacube.drivers.driver_cache WARNING Error was: VersionConflict(rasterio 1.2.10 (/home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages), Requirement.parse('rasterio>=1.3.2'))
2022-12-10 23:35:49,774 512681 datacube.drivers.driver_cache WARNING Failed to resolve driver datacube.plugins.index::null
2022-12-10 23:35:49,774 512681 datacube.drivers.driver_cache WARNING Error was: VersionConflict(rasterio 1.2.10 (/home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages), Requirement.parse('rasterio>=1.3.2'))
2022-12-10 23:35:49,774 512681 datacube.drivers.driver_cache WARNING Failed to resolve driver datacube.plugins.index::postgis
2022-12-10 23:35:49,774 512681 datacube.drivers.driver_cache WARNING Error was: VersionConflict(rasterio 1.2.10 (/home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages), Requirement.parse('rasterio>=1.3.2'))
YES
dap@odc ~/proj/datacube-core %

Is this success? It does offer 'YES' at the end, but I am a bit concerned, mainly by all the self-test errors. This is a small extract of the verbiage emitted by running the ./check-code.sh integration_tests step.

---------- coverage: platform linux, python 3.8.15-final-0 -----------
Name                                         Stmts   Miss  Cover
----------------------------------------------------------------
datacube/__init__.py                             8      0   100%
datacube/__main__.py                             0      0   100%
datacube/api/__init__.py                         4      0   100%
datacube/api/core.py                           384    107    72%
datacube/api/grid_workflow.py                  137     10    93%
datacube/api/query.py                          213     18    92%
datacube/config.py                             126      3    98%
datacube/drivers/__init__.py                     5      0   100%
datacube/drivers/_tools.py                      14      0   100%
datacube/drivers/_types.py                      46      0   100%
datacube/drivers/datasource.py                  30      0   100%
datacube/drivers/driver_cache.py                29     10    66%
datacube/drivers/indexes.py                     24      0   100%
datacube/drivers/netcdf/__init__.py              4      0   100%
datacube/drivers/netcdf/_safestrings.py         41      2    95%
datacube/drivers/netcdf/_write.py               55      0   100%
datacube/drivers/netcdf/driver.py               36     11    69%
datacube/drivers/netcdf/writer.py              168     14    92%
datacube/drivers/postgis/__init__.py             4      0   100%
datacube/drivers/postgis/_api.py               396    288    27%
datacube/drivers/postgis/_connections.py       132     56    58%
datacube/drivers/postgis/_core.py               99     65    34%
datacube/drivers/postgis/_fields.py            266    134    50%
datacube/drivers/postgis/_schema.py            107      1    99%
datacube/drivers/postgis/_spatial.py            88     58    34%
datacube/drivers/postgis/sql.py                 55     12    78%
datacube/drivers/postgres/__init__.py            4      0   100%
datacube/drivers/postgres/_api.py              302     10    97%
datacube/drivers/postgres/_connections.py      105     12    89%
datacube/drivers/postgres/_core.py             107      8    93%
datacube/drivers/postgres/_dynamic.py           64      7    89%
datacube/drivers/postgres/_fields.py           268     25    91%
datacube/drivers/postgres/_schema.py            14      0   100%
datacube/drivers/postgres/sql.py                55      1    98%
datacube/drivers/readers.py                     41      7    83%
datacube/drivers/rio/__init__.py                 1      0   100%
datacube/drivers/rio/_reader.py                134      0   100%
datacube/drivers/writers.py                     20      3    85%
datacube/execution/__init__.py                   0      0   100%
datacube/execution/worker.py                    30     20    33%
datacube/executor.py                           169     59    65%
datacube/helpers.py                             19     14    26%
datacube/index/__init__.py                       6      0   100%
datacube/index/_api.py                          14      1    93%
datacube/index/abstract.py                     378     64    83%
datacube/index/eo3.py                          101      1    99%
datacube/index/exceptions.py                    10      0   100%
datacube/index/fields.py                        31      1    97%
datacube/index/hl.py                           158      1    99%
datacube/index/memory/__init__.py                1      0   100%
datacube/index/memory/_datasets.py             475    409    14%
datacube/index/memory/_fields.py                11      5    55%
datacube/index/memory/_metadata_types.py        69     46    33%
datacube/index/memory/_products.py             107     85    21%
datacube/index/memory/_users.py                 38     27    29%
datacube/index/memory/index.py                  67     25    63%
datacube/index/null/__init__.py                  1      0   100%
datacube/index/null/_datasets.py                63     24    62%
datacube/index/null/_metadata_types.py          16      4    75%
datacube/index/null/_products.py                23      7    70%
datacube/index/null/_users.py                   10      2    80%
datacube/index/null/index.py                    62     22    65%
datacube/index/postgis/__init__.py               0      0   100%
datacube/index/postgis/_datasets.py            377    310    18%
datacube/index/postgis/_metadata_types.py       82     58    29%
datacube/index/postgis/_products.py            126    100    21%
datacube/index/postgis/_transaction.py          27     12    56%
datacube/index/postgis/_users.py                21     11    48%
datacube/index/postgis/index.py                103     52    50%
datacube/index/postgres/__init__.py              0      0   100%
datacube/index/postgres/_datasets.py           366     20    95%
datacube/index/postgres/_metadata_types.py      82      3    96%
datacube/index/postgres/_products.py           123      6    95%
datacube/index/postgres/_transaction.py         27      0   100%
datacube/index/postgres/_users.py               21      0   100%
datacube/index/postgres/index.py                90      1    99%
datacube/model/__init__.py                     515     38    93%
datacube/model/_base.py                          6      0   100%
datacube/model/fields.py                        83      4    95%
datacube/model/utils.py                        164     34    79%
datacube/scripts/__init__.py                     0      0   100%
datacube/scripts/cli_app.py                      8      0   100%
datacube/scripts/dataset.py                    358     38    89%
datacube/scripts/ingest.py                     265    188    29%
datacube/scripts/metadata.py                    95     20    79%
datacube/scripts/product.py                    131     24    82%
datacube/scripts/search_tool.py                 75      2    97%
datacube/scripts/system.py                      54      6    89%
datacube/scripts/user.py                        62      7    89%
datacube/storage/__init__.py                     5      0   100%
datacube/storage/_base.py                       56      0   100%
datacube/storage/_hdf5.py                        2      0   100%
datacube/storage/_load.py                       86      0   100%
datacube/storage/_read.py                      127      3    98%
datacube/storage/_rio.py                       143     15    90%
datacube/storage/masking.py                      3      0   100%
datacube/testutils/__init__.py                 208      4    98%
datacube/testutils/geom.py                      66      0   100%
datacube/testutils/io.py                       204      7    97%
datacube/testutils/iodriver.py                  31      0   100%
datacube/testutils/threads.py                   15      0   100%
datacube/ui/__init__.py                          5      0   100%
datacube/ui/click.py                           163     26    84%
datacube/ui/common.py                           52      1    98%
datacube/ui/expression.py                       46     10    78%
datacube/ui/task_app.py                        159     30    81%
datacube/utils/__init__.py                      10      0   100%
datacube/utils/_misc.py                          7      0   100%
datacube/utils/aws/__init__.py                 180      0   100%
datacube/utils/changes.py                       75      1    99%
datacube/utils/cog.py                          104      0   100%
datacube/utils/dask.py                          93      0   100%
datacube/utils/dates.py                         69      8    88%
datacube/utils/documents.py                    280      5    98%
datacube/utils/generic.py                       39      0   100%
datacube/utils/geometry/__init__.py              5      0   100%
datacube/utils/geometry/_base.py               765     13    98%
datacube/utils/geometry/_warp.py                47      0   100%
datacube/utils/geometry/gbox.py                109      1    99%
datacube/utils/geometry/tools.py               269      0   100%
datacube/utils/io.py                            31      1    97%
datacube/utils/masking.py                      118      0   100%
datacube/utils/math.py                         116      0   100%
datacube/utils/py.py                            33      0   100%
datacube/utils/rio/__init__.py                   3      0   100%
datacube/utils/rio/_rio.py                      65      0   100%
datacube/utils/serialise.py                     44      0   100%
datacube/utils/uris.py                         108      5    95%
datacube/utils/xarray_geoextensions.py         102      0   100%
datacube/virtual/__init__.py                    89     12    87%
datacube/virtual/catalog.py                     37     11    70%
datacube/virtual/expr.py                        47      3    94%
datacube/virtual/impl.py                       449     72    84%
datacube/virtual/transformations.py            213     41    81%
datacube/virtual/utils.py                       31      4    87%
----------------------------------------------------------------
TOTAL                                        13615   2886    79%

======================================================= slowest 5 durations ========================================================
12.37s call     integration_tests/test_config_tool.py::test_add_example_dataset_types[datacube-US/Pacific]
4.16s call     tests/test_utils_aws.py::test_s3_basics
2.05s call     tests/test_concurrent_executor.py::test_concurrent_executor
1.50s call     tests/test_utils_dask.py::test_pmap
1.24s call     tests/test_utils_dask.py::test_compute_tasks
===================================================== short test summary info ======================================================
SKIPPED [2] integration_tests/test_3d.py:26: could not import 'dcio_example.xarray_3d': No module named 'dcio_example'
SKIPPED [1] ../../anaconda3/envs/odc_3.8/lib/python3.8/site-packages/_pytest/doctest.py:452: all tests skipped by +SKIP option
XFAIL tests/test_geometry.py::test_lonalt_bounds_more_than_180
  Bounds computation for large geometries in safe mode is broken
XFAIL tests/test_utils_docs.py::test_merge_with_nan
  Merging dictionaries with content of NaN doesn't work currently
ERROR tests/test_utils_docs.py::test_read_docs_from_http
ERROR tests/ui/test_common.py::test_ui_path_doc_stream
ERROR integration_tests/test_cli_output.py::test_cli_product_subcommand[experimental-US/Pacific] - sqlalchemy.exc.OperationalErro...
ERROR integration_tests/test_cli_output.py::test_cli_product_subcommand[experimental-UTC] - sqlalchemy.exc.OperationalError: (psy...
ERROR integration_tests/test_cli_output.py::test_cli_metadata_subcommand[experimental-US/Pacific] - sqlalchemy.exc.OperationalErr...
ERROR integration_tests/test_cli_output.py::test_cli_metadata_subcommand[experimental-UTC] - sqlalchemy.exc.OperationalError: (ps...
ERROR integration_tests/test_cli_output.py::test_cli_dataset_subcommand[experimental-US/Pacific] - sqlalchemy.exc.OperationalErro...
ERROR integration_tests/test_cli_output.py::test_cli_dataset_subcommand[experimental-UTC] - sqlalchemy.exc.OperationalError: (psy...
ERROR integration_tests/test_cli_output.py::test_readd_and_update_metadata_product_dataset_command[experimental-US/Pacific] - sql...
ERROR integration_tests/test_cli_output.py::test_readd_and_update_metadata_product_dataset_command[experimental-UTC] - sqlalchemy...
ERROR integration_tests/test_config_tool.py::test_add_example_dataset_types[experimental-US/Pacific] - sqlalchemy.exc.Operational...
ERROR integration_tests/test_config_tool.py::test_add_example_dataset_types[experimental-UTC] - sqlalchemy.exc.OperationalError: ...
ERROR integration_tests/test_config_tool.py::test_error_returned_on_invalid[experimental-US/Pacific] - sqlalchemy.exc.Operational...
ERROR integration_tests/test_config_tool.py::test_error_returned_on_invalid[experimental-UTC] - sqlalchemy.exc.OperationalError: ...
ERROR integration_tests/test_config_tool.py::test_config_check[experimental-US/Pacific] - sqlalchemy.exc.OperationalError: (psyco...
ERROR integration_tests/test_config_tool.py::test_config_check[experimental-UTC] - sqlalchemy.exc.OperationalError: (psycopg2.Ope...
ERROR integration_tests/test_config_tool.py::test_list_users_does_not_fail[experimental-US/Pacific] - sqlalchemy.exc.OperationalE...
ERROR integration_tests/test_config_tool.py::test_list_users_does_not_fail[experimental-UTC] - sqlalchemy.exc.OperationalError: (...
ERROR integration_tests/test_config_tool.py::test_db_init_noop[experimental-US/Pacific] - sqlalchemy.exc.OperationalError: (psyco...
ERROR integration_tests/test_config_tool.py::test_db_init_noop[experimental-UTC] - sqlalchemy.exc.OperationalError: (psycopg2.Ope...
ERROR integration_tests/test_config_tool.py::test_db_init[experimental-US/Pacific] - sqlalchemy.exc.OperationalError: (psycopg2.O...
ERROR integration_tests/test_config_tool.py::test_db_init[experimental-UTC] - sqlalchemy.exc.OperationalError: (psycopg2.Operatio...
ERROR integration_tests/test_config_tool.py::test_add_no_such_product[experimental-US/Pacific] - sqlalchemy.exc.OperationalError:...
ERROR integration_tests/test_config_tool.py::test_add_no_such_product[experimental-UTC] - sqlalchemy.exc.OperationalError: (psyco...
ERROR integration_tests/test_config_tool.py::test_user_creation[experimental-example_user0-US/Pacific] - sqlalchemy.exc.Operation...
ERROR integration_tests/test_config_tool.py::test_user_creation[experimental-example_user0-UTC] - sqlalchemy.exc.OperationalError...
ERROR integration_tests/test_config_tool.py::test_user_creation[experimental-example_user1-US/Pacific] - sqlalchemy.exc.Operation...
ERROR integration_tests/test_config_tool.py::test_user_creation[experimental-example_user1-UTC] - sqlalchemy.exc.OperationalError...
ERROR integration_tests/test_config_tool.py::test_user_creation[experimental-example_user2-US/Pacific] - sqlalchemy.exc.Operation...
ERROR integration_tests/test_config_tool.py::test_user_creation[experimental-example_user2-UTC] - sqlalchemy.exc.OperationalError...
ERROR integration_tests/test_config_tool.py::test_user_creation[experimental-example_user3-US/Pacific] - sqlalchemy.exc.Operation...
ERROR integration_tests/test_config_tool.py::test_user_creation[experimental-example_user3-UTC] - sqlalchemy.exc.OperationalError...
ERROR integration_tests/test_dataset_add.py::test_dataset_add_http[US/Pacific-datacube]
ERROR integration_tests/test_dataset_add.py::test_dataset_add_http[UTC-datacube]
ERROR integration_tests/test_model.py::test_crs_parse[experimental-US/Pacific] - sqlalchemy.exc.OperationalError: (psycopg2.Opera...
ERROR integration_tests/test_model.py::test_crs_parse[experimental-UTC] - sqlalchemy.exc.OperationalError: (psycopg2.OperationalE...
ERROR integration_tests/test_validate_ingestion.py::test_invalid_ingestor_config[experimental-US/Pacific] - sqlalchemy.exc.Operat...
ERROR integration_tests/test_validate_ingestion.py::test_invalid_ingestor_config[experimental-UTC] - sqlalchemy.exc.OperationalEr...
ERROR integration_tests/index/test_config_docs.py::test_idempotent_add_dataset_type[experimental-US/Pacific] - sqlalchemy.exc.Ope...
ERROR integration_tests/index/test_config_docs.py::test_idempotent_add_dataset_type[experimental-UTC] - sqlalchemy.exc.Operationa...
ERROR integration_tests/index/test_config_docs.py::test_update_dataset[experimental-US/Pacific] - sqlalchemy.exc.OperationalError...
ERROR integration_tests/index/test_config_docs.py::test_update_dataset[experimental-UTC] - sqlalchemy.exc.OperationalError: (psyc...
ERROR integration_tests/index/test_config_docs.py::test_product_update_cli[experimental-US/Pacific] - sqlalchemy.exc.OperationalE...
ERROR integration_tests/index/test_config_docs.py::test_product_update_cli[experimental-UTC] - sqlalchemy.exc.OperationalError: (...
ERROR integration_tests/index/test_config_docs.py::test_update_metadata_type[experimental-US/Pacific] - sqlalchemy.exc.Operationa...
ERROR integration_tests/index/test_config_docs.py::test_update_metadata_type[experimental-UTC] - sqlalchemy.exc.OperationalError:...
ERROR integration_tests/index/test_config_docs.py::test_filter_types_by_fields[experimental-US/Pacific] - sqlalchemy.exc.Operatio...
ERROR integration_tests/index/test_config_docs.py::test_filter_types_by_fields[experimental-UTC] - sqlalchemy.exc.OperationalErro...
ERROR integration_tests/index/test_config_docs.py::test_filter_types_by_search[experimental-US/Pacific] - sqlalchemy.exc.Operatio...
ERROR integration_tests/index/test_config_docs.py::test_filter_types_by_search[experimental-UTC] - sqlalchemy.exc.OperationalErro...
ERROR integration_tests/index/test_index_data.py::test_archive_datasets[experimental-US/Pacific] - sqlalchemy.exc.OperationalErro...
ERROR integration_tests/index/test_index_data.py::test_archive_datasets[experimental-UTC] - sqlalchemy.exc.OperationalError: (psy...
ERROR integration_tests/index/test_index_data.py::test_purge_datasets[experimental-US/Pacific] - sqlalchemy.exc.OperationalError:...
ERROR integration_tests/index/test_index_data.py::test_purge_datasets[experimental-UTC] - sqlalchemy.exc.OperationalError: (psyco...
ERROR integration_tests/index/test_index_data.py::test_purge_datasets_cli[experimental-US/Pacific] - sqlalchemy.exc.OperationalEr...
ERROR integration_tests/index/test_index_data.py::test_purge_datasets_cli[experimental-UTC] - sqlalchemy.exc.OperationalError: (p...
ERROR integration_tests/index/test_index_data.py::test_purge_all_datasets_cli[experimental-US/Pacific] - sqlalchemy.exc.Operation...
ERROR integration_tests/index/test_index_data.py::test_purge_all_datasets_cli[experimental-UTC] - sqlalchemy.exc.OperationalError...
ERROR integration_tests/index/test_index_data.py::test_index_duplicate_dataset[experimental-US/Pacific] - sqlalchemy.exc.Operatio...
ERROR integration_tests/index/test_index_data.py::test_index_duplicate_dataset[experimental-UTC] - sqlalchemy.exc.OperationalErro...
ERROR integration_tests/index/test_index_data.py::test_has_dataset[experimental-US/Pacific] - sqlalchemy.exc.OperationalError: (p...
ERROR integration_tests/index/test_index_data.py::test_has_dataset[experimental-UTC] - sqlalchemy.exc.OperationalError: (psycopg2...
ERROR integration_tests/index/test_index_data.py::test_get_dataset[experimental-US/Pacific] - sqlalchemy.exc.OperationalError: (p...
ERROR integration_tests/index/test_index_data.py::test_get_dataset[experimental-UTC] - sqlalchemy.exc.OperationalError: (psycopg2...
ERROR integration_tests/index/test_index_data.py::test_transactions_api_ctx_mgr[experimental-US/Pacific] - sqlalchemy.exc.Operati...
ERROR integration_tests/index/test_index_data.py::test_transactions_api_ctx_mgr[experimental-UTC] - sqlalchemy.exc.OperationalErr...
ERROR integration_tests/index/test_index_data.py::test_transactions_api_manual[experimental-US/Pacific] - sqlalchemy.exc.Operatio...
ERROR integration_tests/index/test_index_data.py::test_transactions_api_manual[experimental-UTC] - sqlalchemy.exc.OperationalErro...
ERROR integration_tests/index/test_index_data.py::test_transactions_api_hybrid[experimental-US/Pacific] - sqlalchemy.exc.Operatio...
ERROR integration_tests/index/test_index_data.py::test_transactions_api_hybrid[experimental-UTC] - sqlalchemy.exc.OperationalErro...
ERROR integration_tests/index/test_index_data.py::test_get_missing_things[experimental-US/Pacific] - sqlalchemy.exc.OperationalEr...
ERROR integration_tests/index/test_index_data.py::test_get_missing_things[experimental-UTC] - sqlalchemy.exc.OperationalError: (p...
ERROR integration_tests/index/test_memory_index.py::test_mem_user_resource - RuntimeError: No index driver found for 'memory'. 2 ...
ERROR integration_tests/index/test_memory_index.py::test_mem_metadatatype_resource - RuntimeError: No index driver found for 'mem...
ERROR integration_tests/index/test_memory_index.py::test_mem_product_resource - RuntimeError: No index driver found for 'memory'....
ERROR integration_tests/index/test_memory_index.py::test_mem_dataset_add_eo3 - RuntimeError: No index driver found for 'memory'. ...
ERROR integration_tests/index/test_memory_index.py::test_mem_ds_lineage - RuntimeError: No index driver found for 'memory'. 2 ava...
ERROR integration_tests/index/test_memory_index.py::test_mem_ds_search_dups - RuntimeError: No index driver found for 'memory'. 2...
ERROR integration_tests/index/test_memory_index.py::test_mem_ds_locations - RuntimeError: No index driver found for 'memory'. 2 a...
ERROR integration_tests/index/test_memory_index.py::test_mem_ds_updates - RuntimeError: No index driver found for 'memory'. 2 ava...
ERROR integration_tests/index/test_memory_index.py::test_mem_ds_expand_periods - RuntimeError: No index driver found for 'memory'...
ERROR integration_tests/index/test_memory_index.py::test_mem_prod_time_bounds - RuntimeError: No index driver found for 'memory'....
ERROR integration_tests/index/test_memory_index.py::test_mem_ds_archive_purge - RuntimeError: No index driver found for 'memory'....
ERROR integration_tests/index/test_memory_index.py::test_mem_ds_search_and_count - RuntimeError: No index driver found for 'memor...
ERROR integration_tests/index/test_memory_index.py::test_mem_ds_search_and_count_by_product - RuntimeError: No index driver found...
ERROR integration_tests/index/test_memory_index.py::test_mem_ds_search_returning - RuntimeError: No index driver found for 'memor...
ERROR integration_tests/index/test_memory_index.py::test_mem_ds_search_summary - RuntimeError: No index driver found for 'memory'...
ERROR integration_tests/index/test_memory_index.py::test_mem_ds_search_returning_datasets_light - RuntimeError: No index driver f...
ERROR integration_tests/index/test_memory_index.py::test_mem_ds_search_by_metadata - RuntimeError: No index driver found for 'mem...
ERROR integration_tests/index/test_memory_index.py::test_mem_ds_count_product_through_time - RuntimeError: No index driver found ...
ERROR integration_tests/index/test_memory_index.py::test_memory_dataset_add - RuntimeError: No index driver found for 'memory'. 2...
ERROR integration_tests/index/test_memory_index.py::test_mem_transactions - RuntimeError: No index driver found for 'memory'. 2 a...
ERROR integration_tests/index/test_pluggable_indexes.py::test_with_standard_index[experimental-US/Pacific] - sqlalchemy.exc.Opera...
ERROR integration_tests/index/test_pluggable_indexes.py::test_with_standard_index[experimental-UTC] - sqlalchemy.exc.OperationalE...
ERROR integration_tests/index/test_pluggable_indexes.py::test_system_init[experimental-US/Pacific] - sqlalchemy.exc.OperationalEr...
ERROR integration_tests/index/test_pluggable_indexes.py::test_system_init[experimental-UTC] - sqlalchemy.exc.OperationalError: (p...
ERROR integration_tests/index/test_postgis_index.py::test_create_spatial_index[US/Pacific-experimental] - sqlalchemy.exc.Operatio...
ERROR integration_tests/index/test_postgis_index.py::test_create_spatial_index[UTC-experimental] - sqlalchemy.exc.OperationalErro...
ERROR integration_tests/index/test_postgis_index.py::test_spatial_index_maintain[US/Pacific-experimental] - sqlalchemy.exc.Operat...
ERROR integration_tests/index/test_postgis_index.py::test_spatial_index_maintain[UTC-experimental] - sqlalchemy.exc.OperationalEr...
ERROR integration_tests/index/test_postgis_index.py::test_spatial_index_populate[US/Pacific-experimental] - sqlalchemy.exc.Operat...
ERROR integration_tests/index/test_postgis_index.py::test_spatial_index_populate[UTC-experimental] - sqlalchemy.exc.OperationalEr...
ERROR integration_tests/index/test_postgis_index.py::test_spatial_index_crs_validity[US/Pacific-experimental] - sqlalchemy.exc.Op...
ERROR integration_tests/index/test_postgis_index.py::test_spatial_index_crs_validity[UTC-experimental] - sqlalchemy.exc.Operation...
ERROR integration_tests/index/test_postgis_index.py::test_spatial_extent[US/Pacific-experimental] - sqlalchemy.exc.OperationalErr...
ERROR integration_tests/index/test_postgis_index.py::test_spatial_extent[UTC-experimental] - sqlalchemy.exc.OperationalError: (ps...
ERROR integration_tests/index/test_postgis_index.py::test_spatial_search[US/Pacific-experimental] - sqlalchemy.exc.OperationalErr...
ERROR integration_tests/index/test_postgis_index.py::test_spatial_search[UTC-experimental] - sqlalchemy.exc.OperationalError: (ps...
ERROR integration_tests/index/test_search_eo3.py::test_search_by_metadata[experimental-US/Pacific] - sqlalchemy.exc.OperationalEr...
ERROR integration_tests/index/test_search_eo3.py::test_search_by_metadata[experimental-UTC] - sqlalchemy.exc.OperationalError: (p...
ERROR integration_tests/index/test_search_eo3.py::test_search_dataset_equals_eo3[experimental-US/Pacific] - sqlalchemy.exc.Operat...
ERROR integration_tests/index/test_search_eo3.py::test_search_dataset_equals_eo3[experimental-UTC] - sqlalchemy.exc.OperationalEr...
ERROR integration_tests/index/test_search_eo3.py::test_search_dataset_by_metadata_eo3[experimental-US/Pacific] - sqlalchemy.exc.O...
ERROR integration_tests/index/test_search_eo3.py::test_search_dataset_by_metadata_eo3[experimental-UTC] - sqlalchemy.exc.Operatio...
ERROR integration_tests/index/test_search_eo3.py::test_search_day_eo3[experimental-US/Pacific] - sqlalchemy.exc.OperationalError:...
ERROR integration_tests/index/test_search_eo3.py::test_search_day_eo3[experimental-UTC] - sqlalchemy.exc.OperationalError: (psyco...
ERROR integration_tests/index/test_search_eo3.py::test_search_dataset_ranges_eo3[experimental-US/Pacific] - sqlalchemy.exc.Operat...
ERROR integration_tests/index/test_search_eo3.py::test_search_dataset_ranges_eo3[experimental-UTC] - sqlalchemy.exc.OperationalEr...
ERROR integration_tests/index/test_search_eo3.py::test_zero_width_range_search[experimental-US/Pacific] - sqlalchemy.exc.Operatio...
ERROR integration_tests/index/test_search_eo3.py::test_zero_width_range_search[experimental-UTC] - sqlalchemy.exc.OperationalErro...
ERROR integration_tests/index/test_search_eo3.py::test_search_globally_eo3[experimental-US/Pacific] - sqlalchemy.exc.OperationalE...
ERROR integration_tests/index/test_search_eo3.py::test_search_globally_eo3[experimental-UTC] - sqlalchemy.exc.OperationalError: (...
ERROR integration_tests/index/test_search_eo3.py::test_search_by_product_eo3[experimental-US/Pacific] - sqlalchemy.exc.Operationa...
ERROR integration_tests/index/test_search_eo3.py::test_search_by_product_eo3[experimental-UTC] - sqlalchemy.exc.OperationalError:...
ERROR integration_tests/index/test_search_eo3.py::test_search_limit_eo3[experimental-US/Pacific] - sqlalchemy.exc.OperationalErro...
ERROR integration_tests/index/test_search_eo3.py::test_search_limit_eo3[experimental-UTC] - sqlalchemy.exc.OperationalError: (psy...
ERROR integration_tests/index/test_search_eo3.py::test_search_or_expressions_eo3[experimental-US/Pacific] - sqlalchemy.exc.Operat...
ERROR integration_tests/index/test_search_eo3.py::test_search_or_expressions_eo3[experimental-UTC] - sqlalchemy.exc.OperationalEr...
ERROR integration_tests/index/test_search_eo3.py::test_search_returning_eo3[experimental-US/Pacific] - sqlalchemy.exc.Operational...
ERROR integration_tests/index/test_search_eo3.py::test_search_returning_eo3[experimental-UTC] - sqlalchemy.exc.OperationalError: ...
ERROR integration_tests/index/test_search_eo3.py::test_search_returning_rows_eo3[experimental-US/Pacific] - sqlalchemy.exc.Operat...
ERROR integration_tests/index/test_search_eo3.py::test_search_returning_rows_eo3[experimental-UTC] - sqlalchemy.exc.OperationalEr...
ERROR integration_tests/index/test_search_eo3.py::test_searches_only_type_eo3[experimental-US/Pacific] - sqlalchemy.exc.Operation...
ERROR integration_tests/index/test_search_eo3.py::test_searches_only_type_eo3[experimental-UTC] - sqlalchemy.exc.OperationalError...
ERROR integration_tests/index/test_search_eo3.py::test_search_special_fields_eo3[experimental-US/Pacific] - sqlalchemy.exc.Operat...
ERROR integration_tests/index/test_search_eo3.py::test_search_special_fields_eo3[experimental-UTC] - sqlalchemy.exc.OperationalEr...
ERROR integration_tests/index/test_search_eo3.py::test_search_by_uri_eo3[experimental-US/Pacific] - sqlalchemy.exc.OperationalErr...
ERROR integration_tests/index/test_search_eo3.py::test_search_by_uri_eo3[experimental-UTC] - sqlalchemy.exc.OperationalError: (ps...
ERROR integration_tests/index/test_search_eo3.py::test_search_conflicting_types[experimental-US/Pacific] - sqlalchemy.exc.Operati...
ERROR integration_tests/index/test_search_eo3.py::test_search_conflicting_types[experimental-UTC] - sqlalchemy.exc.OperationalErr...
ERROR integration_tests/index/test_search_eo3.py::test_fetch_all_of_md_type[experimental-US/Pacific] - sqlalchemy.exc.Operational...
ERROR integration_tests/index/test_search_eo3.py::test_fetch_all_of_md_type[experimental-UTC] - sqlalchemy.exc.OperationalError: ...
ERROR integration_tests/index/test_search_eo3.py::test_count_searches[experimental-US/Pacific] - sqlalchemy.exc.OperationalError:...
ERROR integration_tests/index/test_search_eo3.py::test_count_searches[experimental-UTC] - sqlalchemy.exc.OperationalError: (psyco...
ERROR integration_tests/index/test_search_eo3.py::test_count_by_product_searches_eo3[experimental-US/Pacific] - sqlalchemy.exc.Op...
ERROR integration_tests/index/test_search_eo3.py::test_count_by_product_searches_eo3[experimental-UTC] - sqlalchemy.exc.Operation...
ERROR integration_tests/index/test_search_eo3.py::test_count_time_groups[experimental-US/Pacific] - sqlalchemy.exc.OperationalErr...
ERROR integration_tests/index/test_search_eo3.py::test_count_time_groups[experimental-UTC] - sqlalchemy.exc.OperationalError: (ps...
ERROR integration_tests/index/test_search_eo3.py::test_count_time_groups_cli[experimental-US/Pacific] - sqlalchemy.exc.Operationa...
ERROR integration_tests/index/test_search_eo3.py::test_count_time_groups_cli[experimental-UTC] - sqlalchemy.exc.OperationalError:...
ERROR integration_tests/index/test_search_eo3.py::test_search_cli_basic[experimental-US/Pacific] - sqlalchemy.exc.OperationalErro...
ERROR integration_tests/index/test_search_eo3.py::test_search_cli_basic[experimental-UTC] - sqlalchemy.exc.OperationalError: (psy...
ERROR integration_tests/index/test_search_eo3.py::test_cli_info_eo3[experimental-US/Pacific] - sqlalchemy.exc.OperationalError: (...
ERROR integration_tests/index/test_search_eo3.py::test_cli_info_eo3[experimental-UTC] - sqlalchemy.exc.OperationalError: (psycopg...
ERROR integration_tests/index/test_search_eo3.py::test_find_duplicates_eo3[experimental-US/Pacific] - sqlalchemy.exc.OperationalE...
ERROR integration_tests/index/test_search_eo3.py::test_find_duplicates_eo3[experimental-UTC] - sqlalchemy.exc.OperationalError: (...
ERROR integration_tests/index/test_search_eo3.py::test_csv_search_via_cli_eo3[experimental-US/Pacific] - sqlalchemy.exc.Operation...
ERROR integration_tests/index/test_search_eo3.py::test_csv_search_via_cli_eo3[experimental-UTC] - sqlalchemy.exc.OperationalError...
ERROR integration_tests/index/test_search_eo3.py::test_csv_structure_eo3[experimental-US/Pacific] - sqlalchemy.exc.OperationalErr...
ERROR integration_tests/index/test_search_eo3.py::test_csv_structure_eo3[experimental-UTC] - sqlalchemy.exc.OperationalError: (ps...
ERROR integration_tests/index/test_search_eo3.py::test_query_dataset_multi_product_eo3[experimental-US/Pacific] - sqlalchemy.exc....
ERROR integration_tests/index/test_search_eo3.py::test_query_dataset_multi_product_eo3[experimental-UTC] - sqlalchemy.exc.Operati...
FAILED tests/test_driver.py::test_writer_drivers - AssertionError: assert 'netcdf' in []
FAILED tests/test_driver.py::test_index_drivers - AssertionError: assert 'null' in ['default', 'postgres']
FAILED tests/test_geometry.py::test_ops - assert 25.0 == 25.000000000000004
FAILED tests/test_geometry.py::test_crs_compat - assert None == 3577
FAILED tests/api/test_grid_workflow.py::test_gridworkflow_with_time_depth - AssertionError
FAILED tests/api/test_virtual.py::test_aggregate - ValueError: time already exists as coordinate or variable name.
FAILED integration_tests/test_double_ingestion.py::test_double_ingestion[US/Pacific-datacube] - AssertionError: Error for ['inges...
FAILED integration_tests/test_double_ingestion.py::test_double_ingestion[UTC-datacube] - AssertionError: Error for ['ingest', '--...
FAILED integration_tests/test_end_to_end.py::test_end_to_end[US/Pacific-datacube] - AssertionError: Error for ['-v', 'ingest', '-...
FAILED integration_tests/test_end_to_end.py::test_end_to_end[UTC-datacube] - AssertionError: Error for ['-v', 'ingest', '-c', '/t...
FAILED integration_tests/test_full_ingestion.py::test_full_ingestion[US/Pacific-datacube] - AssertionError: Error for ['ingest', ...
FAILED integration_tests/test_full_ingestion.py::test_full_ingestion[UTC-datacube] - AssertionError: Error for ['ingest', '--conf...
FAILED integration_tests/test_full_ingestion.py::test_process_all_ingest_jobs[US/Pacific-datacube] - AssertionError: Error for ['...
FAILED integration_tests/test_full_ingestion.py::test_process_all_ingest_jobs[UTC-datacube] - AssertionError: Error for ['ingest'...
FAILED integration_tests/test_index_out_of_bound.py::test_index_out_of_bound_error[US/Pacific-datacube] - AssertionError: Error f...
FAILED integration_tests/test_index_out_of_bound.py::test_index_out_of_bound_error[UTC-datacube] - AssertionError: Error for ['in...
FAILED integration_tests/test_validate_ingestion.py::test_invalid_ingestor_config[datacube-US/Pacific] - AssertionError: assert '...
FAILED integration_tests/test_validate_ingestion.py::test_invalid_ingestor_config[datacube-UTC] - AssertionError: assert 'No such...
FAILED integration_tests/index/test_memory_index.py::test_init_memory - AssertionError: assert 'memory' in {'default': <datacube....
FAILED integration_tests/index/test_null_index.py::test_init_null - AssertionError: assert 'null' in {'default': <datacube.index....
FAILED integration_tests/index/test_null_index.py::test_null_user_resource - RuntimeError: No index driver found for 'null'. 2 av...
FAILED integration_tests/index/test_null_index.py::test_null_metadata_types_resource - RuntimeError: No index driver found for 'n...
FAILED integration_tests/index/test_null_index.py::test_null_product_resource - RuntimeError: No index driver found for 'null'. 2...
FAILED integration_tests/index/test_null_index.py::test_null_dataset_resource - RuntimeError: No index driver found for 'null'. 2...
FAILED integration_tests/index/test_null_index.py::test_null_transactions - RuntimeError: No index driver found for 'null'. 2 ava...
FAILED integration_tests/index/test_search_eo3.py::test_cli_info_eo3[datacube-US/Pacific] - AssertionError: assert '    lat: {beg...
FAILED integration_tests/index/test_search_eo3.py::test_cli_info_eo3[datacube-UTC] - AssertionError: assert '    lat: {begin: -38...
==================== 27 failed, 690 passed, 3 skipped, 2 xfailed, 21 warnings, 162 errors in 158.71s (0:02:38) =====================

Environment information

Which datacube --version are you using?


dap@odc ~/proj/datacube-core % datacube --version
Open Data Cube core, version 1.8.8
dap@odc ~/proj/datacube-core % conda info

 active environment : odc_3.8
active env location : /home/dap/anaconda3/envs/odc_3.8
        shell level : 1
   user config file : /home/dap/.condarc
populated config files : /home/dap/.condarc
      conda version : 22.11.1
conda-build version : 3.22.0
     python version : 3.9.13.final.0
   virtual packages : __archspec=1=aarch64
                      __glibc=2.35=0
                      __linux=5.15.0=0
                      __unix=0=0
   base environment : /home/dap/anaconda3  (writable)
  conda av data dir : /home/dap/anaconda3/etc/conda
conda av metadata url : None
       channel URLs : https://repo.anaconda.com/pkgs/main/linux-aarch64
                      https://repo.anaconda.com/pkgs/main/noarch
                      https://repo.anaconda.com/pkgs/r/linux-aarch64
                      https://repo.anaconda.com/pkgs/r/noarch
                      https://conda.anaconda.org/conda-forge/linux-aarch64
                      https://conda.anaconda.org/conda-forge/noarch
      package cache : /home/dap/anaconda3/pkgs
                      /home/dap/.conda/pkgs
   envs directories : /home/dap/anaconda3/envs
                      /home/dap/.conda/envs
           platform : linux-aarch64
         user-agent : conda/22.11.1 requests/2.28.1 CPython/3.9.13 Linux/5.15.0-56-generic ubuntu/22.04.1 glibc/2.35
            UID:GID : 1027:1027
         netrc file : None
       offline mode : False

dap@odc ~/proj/datacube-core % git status On branch develop Your branch is up to date with 'origin/develop'.

Untracked files: (use "git add ..." to include in what will be committed) h.txt my.py redo.sh

nothing added to commit but untracked files present (use "git add" to track) dap@odc ~/proj/datacube-core %


* What datacube deployment/enviornment are you running against?

> **Note:** Stale issues will be automatically closed after a period of six months with no activity. 
> To ensure critical issues are not closed, tag them with the Github `pinned` tag.
> If you are a community member and not a maintainer please escalate this issue to maintainers via
> [GIS StackExchange](https://gis.stackexchange.com/questions/tagged/open-data-cube) or [Slack](http://slack.opendatacube.org).

omad commented 1 year ago

Hi permezel,

I'm sorry for the frustrations you must be having, thanks for documenting so clearly what you've tried. The Installation and Setup pages in the ODC documentation are clearly quite out of date.

I think there's a couple of issues you're hitting, which is confusing everything.

Someone, an old version of rasterio has been installed in your environment. Because of the version restrictions ODC has set, this is preventing any of the ODC Drivers from being loaded.

You should be able to fix this by running conda upgrade rasterio.
I can't tell for sure, but it looks like the integration tests can't connect to the database.

By default, they attempt to connect to a DB called agdcintegration, running on a local UNIX socket.

It appears you've overridden this with ~/.datacube_integration.conf, however this may be specifying either a database that doesn't exist, or that can't be connected to for another reason like incorrect credentials or hostname.

Can you please upgrade rasterio, and check the contents of ~/.datacube_integration.conf, including it here if you have any questions.

Cheers Damien

permezel commented 1 year ago

Hi @omad I tried the conda upgrade rasterio and it did some stuff but left me at version 1.2.10, so still the same errors from data cube system check I then tried conda install rasterio=1.3.2 but that resulted in a (familiar) issue of

  File "/home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages/psycopg2/__init__.py", line 51, in <module>
    from psycopg2._psycopg import (                     # noqa
ImportError: /home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages/psycopg2/_psycopg.cpython-38-aarch64-linux-gnu.so: undefined symbol: PQencryptPasswordConn

So I tried conda install rasterio=1.3.4 which is the latest version. This results in

dap@odc ~/proj/datacube-core % datacube system check
Traceback (most recent call last):
  File "/home/dap/anaconda3/envs/odc_3.8/bin/datacube", line 6, in <module>
    from datacube.scripts.cli_app import cli
  File "/home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages/datacube/__init__.py", line 29, in <module>
    from .api import Datacube
  File "/home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages/datacube/api/__init__.py", line 9, in <module>
    from .core import Datacube, TerminateCurrentLoad
  File "/home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages/datacube/api/core.py", line 16, in <module>
    from datacube.storage import reproject_and_fuse, BandInfo
  File "/home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages/datacube/storage/__init__.py", line 11, in <module>
    from ..drivers.datasource import (
  File "/home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages/datacube/drivers/__init__.py", line 9, in <module>
    from .indexes import index_driver_by_name, index_drivers
  File "/home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages/datacube/drivers/indexes.py", line 9, in <module>
    from ..index.abstract import AbstractIndexDriver
  File "/home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages/datacube/index/__init__.py", line 9, in <module>
    from ._api import index_connect
  File "/home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages/datacube/index/_api.py", line 12, in <module>
    from datacube.index.abstract import AbstractIndex
  File "/home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages/datacube/index/abstract.py", line 18, in <module>
    from datacube.index.fields import Field
  File "/home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages/datacube/index/fields.py", line 13, in <module>
    from datacube.model import Range
  File "/home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages/datacube/model/__init__.py", line 20, in <module>
    from datacube.utils import geometry, without_lineage_sources, parse_time, cached_property, uri_to_local_path, \
  File "/home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages/datacube/utils/geometry/__init__.py", line 8, in <module>
    from ._base import (
  File "/home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages/datacube/utils/geometry/_base.py", line 19, in <module>
    import rasterio                    # type: ignore[import]
  File "/home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages/rasterio/__init__.py", line 28, in <module>
    from rasterio._version import gdal_version, get_geos_version, get_proj_version
ImportError: libLerc.so.4: cannot open shared object file: No such file or directory

So I tried conda upgrade --all and I get more or less the same failure as above.

I also tried conda install -c conda-forge lerc to no avail.

  File "/home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages/datacube/utils/geometry/__init__.py", line 8, in <module>
    from ._base import (
  File "/home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages/datacube/utils/geometry/_base.py", line 19, in <module>
    import rasterio                    # type: ignore[import]
  File "/home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages/rasterio/__init__.py", line 28, in <module>
    from rasterio._version import gdal_version, get_geos_version, get_proj_version
ImportError: libLerc.so.4: cannot open shared object file: No such file or directory
dap@odc ~/proj/datacube-core % find ~ -print | grep Lerc
/home/dap/anaconda3/lib/libLerc.so
/home/dap/anaconda3/include/Lerc_types.h
/home/dap/anaconda3/include/Lerc_c_api.h
/home/dap/anaconda3/envs/odc/lib/libLerc.so
/home/dap/anaconda3/envs/odc/include/Lerc_types.h
/home/dap/anaconda3/envs/odc/include/Lerc_c_api.h
/home/dap/anaconda3/envs/odc_3.8/lib/libLerc.so
/home/dap/anaconda3/envs/odc_3.8/include/Lerc_types.h
/home/dap/anaconda3/envs/odc_3.8/include/Lerc_c_api.h
/home/dap/anaconda3/pkgs/lerc-3.0-h01db608_0/lib/libLerc.so
/home/dap/anaconda3/pkgs/lerc-3.0-h01db608_0/include/Lerc_types.h
/home/dap/anaconda3/pkgs/lerc-3.0-h01db608_0/include/Lerc_c_api.h
/home/dap/anaconda3/pkgs/lerc-3.0-h22f4aa5_0/lib/libLerc.so
/home/dap/anaconda3/pkgs/lerc-3.0-h22f4aa5_0/include/Lerc_types.h
/home/dap/anaconda3/pkgs/lerc-3.0-h22f4aa5_0/include/Lerc_c_api.h
dap@odc ~/proj/datacube-core %

So I try:

dap@odc ~/proj/datacube-core % conda install -c conda-forge lerc=4.0.0
...
dap@odc ~/proj/datacube-core % find ~ -print | grep Lerc              
/home/dap/anaconda3/lib/libLerc.so
/home/dap/anaconda3/include/Lerc_types.h
/home/dap/anaconda3/include/Lerc_c_api.h
/home/dap/anaconda3/envs/odc/lib/libLerc.so
/home/dap/anaconda3/envs/odc/include/Lerc_types.h
/home/dap/anaconda3/envs/odc/include/Lerc_c_api.h
/home/dap/anaconda3/envs/odc_3.8/lib/pkgconfig/Lerc.pc
/home/dap/anaconda3/envs/odc_3.8/lib/libLerc.so
/home/dap/anaconda3/envs/odc_3.8/lib/libLerc.so.4
/home/dap/anaconda3/envs/odc_3.8/include/Lerc_types.h
/home/dap/anaconda3/envs/odc_3.8/include/Lerc_c_api.h
/home/dap/anaconda3/pkgs/lerc-4.0.0-h4de3ea5_0/lib/pkgconfig/Lerc.pc
/home/dap/anaconda3/pkgs/lerc-4.0.0-h4de3ea5_0/lib/libLerc.so
/home/dap/anaconda3/pkgs/lerc-4.0.0-h4de3ea5_0/lib/libLerc.so.4
/home/dap/anaconda3/pkgs/lerc-4.0.0-h4de3ea5_0/include/Lerc_types.h
/home/dap/anaconda3/pkgs/lerc-4.0.0-h4de3ea5_0/include/Lerc_c_api.h
/home/dap/anaconda3/pkgs/lerc-3.0-h01db608_0/lib/libLerc.so
/home/dap/anaconda3/pkgs/lerc-3.0-h01db608_0/include/Lerc_types.h
/home/dap/anaconda3/pkgs/lerc-3.0-h01db608_0/include/Lerc_c_api.h
/home/dap/anaconda3/pkgs/lerc-3.0-h22f4aa5_0/lib/libLerc.so
/home/dap/anaconda3/pkgs/lerc-3.0-h22f4aa5_0/include/Lerc_types.h
/home/dap/anaconda3/pkgs/lerc-3.0-h22f4aa5_0/include/Lerc_c_api.h
dap@odc ~/proj/datacube-core %

Sadly, this gets me back to the old, familiar PQencryptPasswordConn error.

dap@odc ~/proj/datacube-core % datacube system check                  
Traceback (most recent call last):
  File "/home/dap/anaconda3/envs/odc_3.8/bin/datacube", line 6, in <module>
    from datacube.scripts.cli_app import cli
  File "/home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages/datacube/scripts/cli_app.py", line 17, in <module>
    import datacube.scripts.system    # noqa: F401
  File "/home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages/datacube/scripts/system.py", line 13, in <module>
    from datacube.drivers.postgres._connections import IndexSetupError
  File "/home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages/datacube/drivers/postgres/__init__.py", line 11, in <module>
    from ._connections import PostgresDb
  File "/home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages/datacube/drivers/postgres/_connections.py", line 30, in <module>
    from . import _api
  File "/home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages/datacube/drivers/postgres/_api.py", line 31, in <module>
    from ._fields import parse_fields, Expression, PgField, PgExpression  # noqa: F401
  File "/home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages/datacube/drivers/postgres/_fields.py", line 13, in <module>
    from psycopg2.extras import NumericRange, DateTimeTZRange
  File "/home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages/psycopg2/__init__.py", line 51, in <module>
    from psycopg2._psycopg import (                     # noqa
ImportError: /home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages/psycopg2/_psycopg.cpython-38-aarch64-linux-gnu.so: undefined symbol: PQencryptPasswordConn

permezel commented 1 year ago

Depending on what versions of stuff I force into my odc_3.8 env, I occasionally get an installed version of psql. When this happens, I get the (reported elsewhere) problems of psql looking in /tmp for the socket.

dap@odc ~/proj/datacube-core % which psql               
/home/dap/anaconda3/envs/odc_3.8/bin/psql
dap@odc ~/proj/datacube-core % psql -d agdcintegration             
psql: could not connect to server: No such file or directory
    Is the server running locally and accepting
    connections on Unix domain socket "/tmp/.s.PGSQL.5432"?

If I explicitly use the system version, it works:

dap@odc ~/proj/datacube-core % /usr/bin/psql -d agdcintegration
psql (14.5 (Ubuntu 14.5-0ubuntu0.22.04.1))
Type "help" for help.

agdcintegration=# \q
dap@odc ~/proj/datacube-core %

Also, depending on what versions of that are tripping over each other, I either get a working version go psycopg2 or not. Perhaps I also have to force that version.

dap@odc ~/proj/datacube-core % F=$(find ~ -print | grep libpq.so)
for f in $(echo $F); do echo "=-= $f"; nm $f | grep PQenc; done

=-= /home/dap/anaconda3/lib/libpq.so.5.12
000000000000c684 T PQencryptPassword
000000000000c714 T PQencryptPasswordConn
=-= /home/dap/anaconda3/lib/libpq.so.5
000000000000c684 T PQencryptPassword
000000000000c714 T PQencryptPasswordConn
=-= /home/dap/anaconda3/lib/libpq.so
000000000000c684 T PQencryptPassword
000000000000c714 T PQencryptPasswordConn
=-= /home/dap/anaconda3/envs/odc_3.8/lib/libpq.so.5.14
0000000000025470 T PQencryptPassword
0000000000025500 T PQencryptPasswordConn
=-= /home/dap/anaconda3/envs/odc_3.8/lib/libpq.so.5
000000000000b520 T PQencryptPassword
=-= /home/dap/anaconda3/envs/odc_3.8/lib/libpq.so.5.9
000000000000b520 T PQencryptPassword
=-= /home/dap/anaconda3/envs/odc_3.8/lib/libpq.so
000000000000b520 T PQencryptPassword
=-= /home/dap/anaconda3/pkgs/postgresql-9.6.6-he3421e9_1/lib/libpq.so.5
000000000000b520 T PQencryptPassword
=-= /home/dap/anaconda3/pkgs/postgresql-9.6.6-he3421e9_1/lib/libpq.so.5.9
000000000000b520 T PQencryptPassword
=-= /home/dap/anaconda3/pkgs/postgresql-9.6.6-he3421e9_1/lib/libpq.so
000000000000b520 T PQencryptPassword
=-= /home/dap/anaconda3/pkgs/libpq-14.5-hfbf4865_3/lib/libpq.so.5.14
0000000000025470 T PQencryptPassword
0000000000025500 T PQencryptPasswordConn
=-= /home/dap/anaconda3/pkgs/libpq-14.5-hfbf4865_3/lib/libpq.so.5
0000000000025470 T PQencryptPassword
0000000000025500 T PQencryptPasswordConn
=-= /home/dap/anaconda3/pkgs/libpq-14.5-hfbf4865_3/lib/libpq.so
0000000000025470 T PQencryptPassword
0000000000025500 T PQencryptPasswordConn
=-= /home/dap/anaconda3/pkgs/libpq-14.5-h0f47c37_2/lib/libpq.so.5.14
0000000000025460 T PQencryptPassword
00000000000254f0 T PQencryptPasswordConn
=-= /home/dap/anaconda3/pkgs/libpq-14.5-h0f47c37_2/lib/libpq.so.5
0000000000025460 T PQencryptPassword
00000000000254f0 T PQencryptPasswordConn
=-= /home/dap/anaconda3/pkgs/libpq-14.5-h0f47c37_2/lib/libpq.so
0000000000025460 T PQencryptPassword
00000000000254f0 T PQencryptPasswordConn
=-= /home/dap/anaconda3/pkgs/libpq-12.9-h140f9b7_3/lib/libpq.so.5.12
000000000000c684 T PQencryptPassword
000000000000c714 T PQencryptPasswordConn
=-= /home/dap/anaconda3/pkgs/libpq-12.9-h140f9b7_3/lib/libpq.so.5
000000000000c684 T PQencryptPassword
000000000000c714 T PQencryptPasswordConn
=-= /home/dap/anaconda3/pkgs/libpq-12.9-h140f9b7_3/lib/libpq.so
000000000000c684 T PQencryptPassword
000000000000c714 T PQencryptPasswordConn
dap@odc ~/proj/datacube-core %

Anaconda does not seem capable of correctly managing the dependencies. Perhaps a pyenv and pip based approach would be more fruitful? Alternately, although my mind boggles at the thought of trying to mess with nix, it does claim to be able to correctly manage dependencies in a rather spiffy manner.

omad commented 1 year ago

@permezel

I tried the conda upgrade rasterio and it did some stuff but left me at version 1.2.10, so still the same errors from data cube system check I then tried conda install rasterio=1.3.2 but that resulted in a (familiar) issue of
  File "/home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages/psycopg2/__init__.py", line 51, in <module>
    from psycopg2._psycopg import (                     # noqa
ImportError: /home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages/psycopg2/_psycopg.cpython-38-aarch64-linux-gnu.so: undefined symbol: PQencryptPasswordConn

Oooh, what sort of a machine are you running on? That aarch64 bit makes it look like an ARM based system. I don't know of anyone using Linux on ARM with ODC/conda packages/GDAL etc. That might be the root of all your troubles.

permezel commented 1 year ago

@permezel
I tried the conda upgrade rasterio and it did some stuff but left me at version 1.2.10, so still the same errors from data cube system check I then tried conda install rasterio=1.3.2 but that resulted in a (familiar) issue of
  File "/home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages/psycopg2/__init__.py", line 51, in <module>
    from psycopg2._psycopg import (                     # noqa
ImportError: /home/dap/anaconda3/envs/odc_3.8/lib/python3.8/site-packages/psycopg2/_psycopg.cpython-38-aarch64-linux-gnu.so: undefined symbol: PQencryptPasswordConn
Oooh, what sort of a machine are you running on? That aarch64 bit makes it look like an ARM based system. I don't know of anyone using Linux on ARM with ODC/conda packages/GDAL etc. That might be the root of all your troubles.

@omad, I am running in a parallels VM on a Mac Studio box. I am keen to assist in getting this to work for aarch64.

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

opendatacube / datacube-core