search

laminlabs / lamindb

A data framework for biology.
https://docs.lamin.ai
Apache License 2.0
126 stars 10 forks source link

NoCredentialsError: Unable to locate credentials in introduction tutorial #2014

Closed Zethson closed 2 weeks ago

Zethson commented 2 weeks ago

Report

Found and reported by @eroell.

ln.UPath("s3://lamindata/iris_studies").view_tree()

results in

---------------------------------------------------------------------------
NoCredentialsError                        Traceback (most recent call last)
Cell In[6], line 1
----> 1 ln.UPath("s3://lamindata/iris_studies").view_tree()

File ~/Documents/lamin_workspace/lamin_venv_oct/lib/python3.11/site-packages/lamindb_setup/core/upath.py:591, in view_tree(path, level, only_dirs, n_max_files_per_dir_and_type, n_max_files, include_paths, skip_suffixes)
    545 def view_tree(
    546     path: Path,
    547     *,
   (...)
    553     skip_suffixes: list[str] | None = None,
    554 ) -> None:
    555     """Print a visual tree structure of files & directories.
    556 
    557     Args:
   (...)
    589         └── raw_feature_bc_matrix.h5
    590     """
--> 591     message, _ = compute_file_tree(
    592         path,
    593         level=level,
    594         only_dirs=only_dirs,
    595         n_max_files=n_max_files,
    596         n_max_files_per_dir_and_type=n_max_files_per_dir_and_type,
    597         include_paths=include_paths,
...
--> 423         raise NoCredentialsError()
    424     datetime_now = datetime.datetime.utcnow()
    425     request.context['timestamp'] = datetime_now.strftime(SIGV4_TIMESTAMP)

NoCredentialsError: Unable to locate credentials

Version information

aiobotocore               2.15.1
aiohappyeyeballs          2.4.3
aiohttp                   3.10.8
aioitertools              0.12.0
aiosignal                 1.3.1
anndata                   0.10.9
annotated-types           0.7.0
anyio                     4.6.0
appdirs                   1.4.4
appnope                   0.1.4
array_api_compat          1.8
asgiref                   3.8.1
asttokens                 2.4.1
attrs                     24.2.0
beautifulsoup4            4.12.3
bionty                    0.51.1
bleach                    6.1.0
boto3                     1.35.23
botocore                  1.35.23
certifi                   2024.8.30
chardet                   5.2.0
charset-normalizer        3.3.2
click                     8.1.7
comm                      0.2.2
debugpy                   1.8.6
decorator                 5.1.1
defusedxml                0.7.1
deprecation               2.1.0
dj-database-url           2.2.0
Django                    5.1.1
executing                 2.1.0
fastjsonschema            2.20.0
fastobo                   0.12.3
filelock                  3.16.1
frozenlist                1.4.1
fsspec                    2024.6.1
gotrue                    2.8.1
graphviz                  0.20.3
h11                       0.14.0
h2                        4.1.0
h5py                      3.12.1
hpack                     4.0.0
httpcore                  0.17.3
httpx                     0.24.1
hyperframe                6.0.1
idna                      3.10
importlib_metadata        8.5.0
iniconfig                 2.0.0
ipykernel                 6.29.5
ipython                   8.27.0
jedi                      0.19.1
Jinja2                    3.1.4
jmespath                  1.0.1
jsonschema                4.23.0
jsonschema-specifications 2023.12.1
jupyter_client            8.6.3
jupyter_core              5.7.2
jupyterlab_pygments       0.3.0
jupytext                  1.16.4
lamin_cli                 0.17.8
lamin_utils               0.13.6
lamindb                   0.76.11     /Users/eljas.roellin/Documents/lamin_workspace/lamindb
lamindb_setup             0.77.7
lnschema_core             0.74.6
markdown-it-py            3.0.0
MarkupSafe                2.1.5
matplotlib-inline         0.1.7
mdit-py-plugins           0.4.2
mdurl                     0.1.2
mistune                   3.0.2
multidict                 6.1.0
natsort                   8.4.0
nbclient                  0.10.0
nbconvert                 7.16.4
nbformat                  5.10.4
nbproject                 0.10.4
nest-asyncio              1.6.0
networkx                  3.3
numpy                     1.26.4
orjson                    3.10.7
packaging                 24.1
pandas                    2.2.3
pandocfilters             1.5.1
parso                     0.8.4
pexpect                   4.9.0
pip                       24.0
platformdirs              4.3.6
pluggy                    1.5.0
postgrest                 0.13.2
prompt_toolkit            3.0.48
pronto                    2.5.5
psutil                    6.0.0
psycopg2-binary           2.9.9
ptyprocess                0.7.0
pure_eval                 0.2.3
pyarrow                   17.0.0
pydantic                  2.9.2
pydantic_core             2.23.4
pydantic-settings         2.5.2
Pygments                  2.18.0
pytest                    8.3.3
python-dateutil           2.9.0.post0
python-dotenv             1.0.1
pytz                      2024.2
PyYAML                    6.0.2
pyzmq                     26.2.0
RapidFuzz                 3.10.0
realtime                  1.0.6
referencing               0.35.1
requests                  2.32.3
rich                      13.9.1
rich-click                1.8.3
rpds-py                   0.20.0
s3fs                      2024.6.1
s3transfer                0.10.2
scipy                     1.12.0
setuptools                65.5.0
six                       1.16.0
sniffio                   1.3.1
soupsieve                 2.6
sqlparse                  0.5.1
stack-data                0.6.3
storage3                  0.7.7
StrEnum                   0.4.15
supabase                  2.2.1
supafunc                  0.3.3
tinycss2                  1.3.0
tornado                   6.4.1
traitlets                 5.14.3
typing_extensions         4.12.2
tzdata                    2024.2
universal_pathlib         0.2.5
urllib3                   1.26.20
wcwidth                   0.2.13
webencodings              0.5.1
websockets                12.0
wrapt                     1.16.0
yarl                      1.13.1
zipp                      3.20.2
Zethson commented 2 weeks ago 
Traceback (most recent call last):
  File "/Users/eljas.roellin/Documents/lamin_workspace/lamin_venv/lib/python3.11/site-packages/lamindb_setup/core/_settings_load.py", line 33, in load_instance_settings
    settings_store = InstanceSettingsStore(_env_file=instance_settings_file)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/eljas.roellin/Documents/lamin_workspace/lamin_venv/lib/python3.11/site-packages/pydantic_settings/main.py", line 144, in __init__
    super().__init__(
  File "/Users/eljas.roellin/Documents/lamin_workspace/lamin_venv/lib/python3.11/site-packages/pydantic/main.py", line 193, in __init__
    self.__pydantic_validator__.validate_python(data, self_instance=self)
pydantic_core._pydantic_core.ValidationError: 2 validation errors for InstanceSettingsStore
lamindb_instance_api_url
  Extra inputs are not permitted [type=extra_forbidden, input_value='null', input_type=str]
    For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden
lamindb_instance_schema_id
  Extra inputs are not permitted [type=extra_forbidden, input_value='null', input_type=str]
    For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/eljas.roellin/Documents/lamin_workspace/lamin_venv/bin/lamin", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/eljas.roellin/Documents/lamin_workspace/lamin_venv/lib/python3.11/site-packages/rich_click/rich_command.py", line 367, in __call__
    return super().__call__(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/eljas.roellin/Documents/lamin_workspace/lamin_venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/eljas.roellin/Documents/lamin_workspace/lamin_venv/lib/python3.11/site-packages/rich_click/rich_command.py", line 152, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/eljas.roellin/Documents/lamin_workspace/lamin_venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/eljas.roellin/Documents/lamin_workspace/lamin_venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/eljas.roellin/Documents/lamin_workspace/lamin_venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/eljas.roellin/Documents/lamin_workspace/lamin_venv/lib/python3.11/site-packages/lamin_cli/__main__.py", line 148, in info
    print(lamindb_setup.settings)
  File "/Users/eljas.roellin/Documents/lamin_workspace/lamin_venv/lib/python3.11/site-packages/lamindb_setup/core/_settings.py", line 124, in __repr__
    if self._instance_exists:
       ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/eljas.roellin/Documents/lamin_workspace/lamin_venv/lib/python3.11/site-packages/lamindb_setup/core/_settings.py", line 113, in _instance_exists
    self.instance  # noqa
    ^^^^^^^^^^^^^
  File "/Users/eljas.roellin/Documents/lamin_workspace/lamin_venv/lib/python3.11/site-packages/lamindb_setup/core/_settings.py", line 101, in instance
    self._instance_settings = load_instance_settings()
                              ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/eljas.roellin/Documents/lamin_workspace/lamin_venv/lib/python3.11/site-packages/lamindb_setup/core/_settings_load.py", line 37, in load_instance_settings
    raise SettingsEnvFileOutdated(
lamindb_setup.core._settings_load.SettingsEnvFileOutdated: 

2 validation errors for InstanceSettingsStore
lamindb_instance_api_url
  Extra inputs are not permitted [type=extra_forbidden, input_value='null', input_type=str]
    For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden
lamindb_instance_schema_id
  Extra inputs are not permitted [type=extra_forbidden, input_value='null', input_type=str]
    For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden

Your instance settings file with

lamindb_instance_api_url=null
lamindb_instance_owner=eroell
lamindb_instance_name=lamin-tutorial
lamindb_instance_storage_root=/Users/eljas.roellin/Documents/lamin_workspace/try_scripts/lamin/tutorials/lamin-tutorial
lamindb_instance_storage_region=null
lamindb_instance_db=null
lamindb_instance_schema_str=null
lamindb_instance_schema_id=null
lamindb_instance_id=f3036c587df25d548671d0c3c02b7867
lamindb_instance_git_repo=null
lamindb_instance_keep_artifacts_local=False

is invalid (likely outdated), see validation error. Please delete /Users/eljas.roellin/.lamin/current_instance.env & reload (remote) or re-initialize (local) the instance with the same name & storage location.

Deleting the file does not seem to help even after restarting the kernel.

Koncopd commented 2 weeks ago

Yes, i guess he doesn't have default aws credentials configured. And ln.UPath is not really aware of our managed credentials. Solutions are:

  1. add some default aws credentials
  2. use create_path - from lamindb_setup.core.upath import create_path; create_path("s3://lamindata/iris_studies").view_tree()
  3. use (ln.setup.storage.root / "iris_studies").view_tree()

I think the tutorial assumes that some credentials are configured. If it is not the case, then maybe we need to expose and use create_path there.

Koncopd commented 2 weeks ago 
2 validation errors for InstanceSettingsStore
lamindb_instance_api_url
  Extra inputs are not permitted [type=extra_forbidden, input_value='null', input_type=str]
    For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden
lamindb_instance_schema_id
  Extra inputs are not permitted [type=extra_forbidden, input_value='null', input_type=str]
    For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden

Hm, interesting... But this is entirely different.

falexwolf commented 2 weeks ago

@eroell should pass anon=True to ln.UPath if he has no AWS setup whatsoever.

@Koncopd will update the tutorial to show this, correct?

Koncopd commented 2 weeks ago

@eroell this should be fixed by this in the tutorial https://github.com/laminlabs/lamindb/pull/2016 Please use ln.UPath("s3://lamindata/iris_studies", anon=True).view_tree()

Zethson commented 2 weeks ago

There was a second environment somehow involved that had an outdated pydantic version. Thanks @Koncopd

eroell commented 2 weeks ago

@eroell should pass anon=True to ln.UPath if he has no AWS setup whatsoever. this helped indeed!