"Incorrect password" when trying to download embargoed file

[rcpeene]

I am running a notebook on dandihub and trying to download an embargoed file that I have access to. Even when providing my dandi api key, I get a prompt to enter a keyring password. I don't know where to find such a password. I don't have this error when running locally or Google collab, just Dandihub.

nwbs = []
file = dandiset.get_asset_by_path(dandi_filepath)
file_url = file.download_url

filename = dandi_filepath.split("/")[-1]
filepath = f"{download_loc}/{filename}"

download.download(file_url, output_dir=download_loc)
print(f"Downloaded file to {filepath}")

print("Opening file")
io = NWBHDF5IO(filepath, mode="r", load_namespaces=True)
nwbs.append(io.read())

And the error (After entering an empty password to the keyring prompt):

---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
File /opt/conda/lib/python3.11/site-packages/dandi/dandiarchive.py:170, in ParsedDandiURL.navigate(self, strict, authenticate)
    169 try:
--> 170     dandiset = self.get_dandiset(client, lazy=not strict)
    171 except requests.HTTPError as e:

File /opt/conda/lib/python3.11/site-packages/dandi/dandiarchive.py:107, in ParsedDandiURL.get_dandiset(self, client, lazy)
    106 if self.dandiset_id is not None:
--> 107     return client.get_dandiset(self.dandiset_id, self.version_id, lazy=lazy)
    108 else:

File /opt/conda/lib/python3.11/site-packages/dandi/dandiapi.py:546, in DandiAPIClient.get_dandiset(self, dandiset_id, version_id, lazy)
    544 try:
    545     d = RemoteDandiset.from_data(
--> 546         self, self.get(f"/dandisets/{dandiset_id}/")
    547     )
    548 except HTTP404Error:

File /opt/conda/lib/python3.11/site-packages/dandi/dandiapi.py:300, in RESTFullAPIClient.get(self, path, **kwargs)
    297 """
    298 Convenience method to call `request()` with the 'GET' HTTP method.
    299 """
--> 300 return self.request("GET", path, **kwargs)

File /opt/conda/lib/python3.11/site-packages/dandi/dandiapi.py:276, in RESTFullAPIClient.request(self, method, path, params, data, files, json, headers, json_resp, retry_statuses, retry_if, **kwargs)
    275     else:
--> 276         raise requests.HTTPError(msg, response=result)
    278 if json_resp:

HTTPError: Error 401 while sending GET request to https://api.dandiarchive.org/api/dandisets/000336/: {"detail":"Authentication credentials were not provided."}

During handling of the above exception, another exception occurred:

AssertionError                            Traceback (most recent call last)
File /opt/conda/lib/python3.11/site-packages/keyrings/alt/file.py:188, in EncryptedKeyring._unlock(self)
    187 try:
--> 188     ref_pw = self.get_password('keyring-setting', 'password reference')
    189     assert ref_pw == 'password reference value'

File /opt/conda/lib/python3.11/site-packages/keyrings/alt/file_base.py:108, in Keyring.get_password(self, service, username)
    107 try:
--> 108     password = self.decrypt(password_encrypted, assoc).decode('utf-8')
    109 except ValueError:
    110     # decrypt the password without associated data

File /opt/conda/lib/python3.11/site-packages/keyrings/alt/file.py:220, in EncryptedKeyring.decrypt(self, password_encrypted, assoc)
    219 plaintext = cipher.decrypt(data['password_encrypted'])
--> 220 assert plaintext.startswith(self.pw_prefix)
    221 return plaintext[3:]

AssertionError: 

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
File /opt/conda/lib/python3.11/site-packages/keyrings/alt/file_base.py:108, in Keyring.get_password(self, service, username)
    107 try:
--> 108     password = self.decrypt(password_encrypted, assoc).decode('utf-8')
    109 except ValueError:
    110     # decrypt the password without associated data

File /opt/conda/lib/python3.11/site-packages/keyrings/alt/file.py:218, in EncryptedKeyring.decrypt(self, password_encrypted, assoc)
    217     data[key] = decodebytes(data[key].encode())
--> 218 cipher = self._create_cipher(self.keyring_key, data['salt'], data['IV'])
    219 plaintext = cipher.decrypt(data['password_encrypted'])

File /opt/conda/lib/python3.11/site-packages/jaraco/classes/properties.py:75, in NonDataProperty.__get__(self, obj, objtype)
     74     return self
---> 75 return self.fget(obj)

File /opt/conda/lib/python3.11/site-packages/keyrings/alt/file.py:100, in EncryptedKeyring.keyring_key(self)
     99 if self._check_file():
--> 100     self._unlock()
    101 else:

File /opt/conda/lib/python3.11/site-packages/keyrings/alt/file.py:192, in EncryptedKeyring._unlock(self)
    191 self._lock()
--> 192 raise ValueError("Incorrect Password")

ValueError: Incorrect Password

During handling of the above exception, another exception occurred:

AssertionError                            Traceback (most recent call last)
File /opt/conda/lib/python3.11/site-packages/keyrings/alt/file.py:188, in EncryptedKeyring._unlock(self)
    187 try:
--> 188     ref_pw = self.get_password('keyring-setting', 'password reference')
    189     assert ref_pw == 'password reference value'

File /opt/conda/lib/python3.11/site-packages/keyrings/alt/file_base.py:108, in Keyring.get_password(self, service, username)
    107 try:
--> 108     password = self.decrypt(password_encrypted, assoc).decode('utf-8')
    109 except ValueError:
    110     # decrypt the password without associated data

File /opt/conda/lib/python3.11/site-packages/keyrings/alt/file.py:220, in EncryptedKeyring.decrypt(self, password_encrypted, assoc)
    219 plaintext = cipher.decrypt(data['password_encrypted'])
--> 220 assert plaintext.startswith(self.pw_prefix)
    221 return plaintext[3:]

AssertionError: 

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
Cell In[10], line 14
     11 filename = dandi_filepath.split("/")[-1]
     12 filepath = f"{download_loc}/{filename}"
---> 14 download.download(file_url, output_dir=download_loc)
     15 print(f"Downloaded file to {filepath}")
     17 print("Opening file")

File /opt/conda/lib/python3.11/site-packages/dandi/download.py:164, in download(urls, output_dir, format, existing, jobs, jobs_per_zarr, get_metadata, get_assets, sync, path_type)
    162 elif format is DownloadFormat.PYOUT:
    163     with out:
--> 164         for rec in gen_:
    165             out(rec)
    166 else:

File /opt/conda/lib/python3.11/site-packages/dandi/download.py:151, in <genexpr>(.0)
    131         lgr.warning(
    132             "Parallel downloads are not yet implemented for non-pyout format=%r. "
    133             "Download will proceed serially.",
    134             str(format),
    135         )
    137 downloaders = [
    138     Downloader(
    139         url=purl,
   (...)
    148     for purl in parsed_urls
    149 ]
--> 151 gen_ = (r for dl in downloaders for r in dl.download_generator())
    153 # TODOs:
    154 #  - redo frontends similarly to how command_ls did it
    155 #  - have a single loop with analysis of `rec` to either any file
    156 #    has failed to download.  If any was: exception should probably be
    157 #    raised.  API discussion for Python side of API:
    158 #
    159 if format is DownloadFormat.DEBUG:

File /opt/conda/lib/python3.11/site-packages/dandi/download.py:235, in Downloader.download_generator(self)
    224 def download_generator(self) -> Iterator[dict]:
    225     """
    226     A generator for downloads of files, folders, or entire dandiset from
    227     DANDI (as identified by URL)
   (...)
    232     schema) while validating their checksums "on the fly", etc.
    233     """
--> 235     with self.url.navigate(strict=True) as (client, dandiset, assets):
    236         if (
    237             isinstance(self.url, DandisetURL)
    238             or (
   (...)
    241             )
    242         ) and self.get_metadata:
    243             assert dandiset is not None

File /opt/conda/lib/python3.11/contextlib.py:137, in _GeneratorContextManager.__enter__(self)
    135 del self.args, self.kwds, self.func
    136 try:
--> 137     return next(self.gen)
    138 except StopIteration:
    139     raise RuntimeError("generator didn't yield") from None

File /opt/conda/lib/python3.11/site-packages/dandi/dandiarchive.py:178, in ParsedDandiURL.navigate(self, strict, authenticate)
    172 if (
    173     e.response is not None
    174     and e.response.status_code == 401
    175     and authenticate is not False
    176 ):
    177     lgr.info("Resource requires authentication; authenticating ...")
--> 178     client.dandi_authenticate()
    179     dandiset = self.get_dandiset(client, lazy=not strict)
    180 else:

File /opt/conda/lib/python3.11/site-packages/dandi/dandiapi.py:495, in DandiAPIClient.dandi_authenticate(self)
    493     return
    494 client_name, app_id = self._get_keyring_ids()
--> 495 keyring_backend, api_key = keyring_lookup(app_id, "key")
    496 key_from_keyring = api_key is not None
    497 while True:

File /opt/conda/lib/python3.11/site-packages/dandi/keyring.py:29, in keyring_lookup(service_name, username)
     22 def keyring_lookup(
     23     service_name: str, username: str
     24 ) -> tuple[KeyringBackend, str | None]:
     25     """
     26     Returns an appropriate keyring backend and the password it holds (if any)
     27     for the given service and username.
     28     """
---> 29     return keyring_op(lambda kb: kb.get_password(service_name, username))

File /opt/conda/lib/python3.11/site-packages/dandi/keyring.py:97, in keyring_op(func)
     95 kb = get_keyring()
     96 try:
---> 97     return (kb, func(kb))
     98 except KeyringError as e:
     99     lgr.info("Default keyring errors on query: %s", e)

File /opt/conda/lib/python3.11/site-packages/dandi/keyring.py:29, in keyring_lookup.<locals>.<lambda>(kb)
     22 def keyring_lookup(
     23     service_name: str, username: str
     24 ) -> tuple[KeyringBackend, str | None]:
     25     """
     26     Returns an appropriate keyring backend and the password it holds (if any)
     27     for the given service and username.
     28     """
---> 29     return keyring_op(lambda kb: kb.get_password(service_name, username))

File /opt/conda/lib/python3.11/site-packages/keyring/backends/chainer.py:48, in ChainerBackend.get_password(self, service, username)
     46 def get_password(self, service, username):
     47     for keyring in self.backends:
---> 48         password = keyring.get_password(service, username)
     49         if password is not None:
     50             return password

File /opt/conda/lib/python3.11/site-packages/keyrings/alt/file_base.py:111, in Keyring.get_password(self, service, username)
    108         password = self.decrypt(password_encrypted, assoc).decode('utf-8')
    109     except ValueError:
    110         # decrypt the password without associated data
--> 111         password = self.decrypt(password_encrypted).decode('utf-8')
    112 except (configparser.NoOptionError, configparser.NoSectionError):
    113     password = None

File /opt/conda/lib/python3.11/site-packages/keyrings/alt/file.py:218, in EncryptedKeyring.decrypt(self, password_encrypted, assoc)
    216 for key in data:
    217     data[key] = decodebytes(data[key].encode())
--> 218 cipher = self._create_cipher(self.keyring_key, data['salt'], data['IV'])
    219 plaintext = cipher.decrypt(data['password_encrypted'])
    220 assert plaintext.startswith(self.pw_prefix)

File /opt/conda/lib/python3.11/site-packages/jaraco/classes/properties.py:75, in NonDataProperty.__get__(self, obj, objtype)
     73 if obj is None:
     74     return self
---> 75 return self.fget(obj)

File /opt/conda/lib/python3.11/site-packages/keyrings/alt/file.py:100, in EncryptedKeyring.keyring_key(self)
     96 @properties.NonDataProperty
     97 def keyring_key(self):
     98     # _unlock or _init_file will set the key or raise an exception
     99     if self._check_file():
--> 100         self._unlock()
    101     else:
    102         self._init_file()

File /opt/conda/lib/python3.11/site-packages/keyrings/alt/file.py:192, in EncryptedKeyring._unlock(self)
    190 except AssertionError:
    191     self._lock()
--> 192     raise ValueError("Incorrect Password")

ValueError: Incorrect Password

[satra]

@jwodder - do you have any suggestions for @rcpeene ? or what to delete to rest the keyring?

[jwodder]

@rcpeene Could you show the (non-exception) output from the script when run, particularly the part that asks you for a password etc.?

Even when providing my dandi api key, I get a prompt to enter a keyring password.

How are you providing the API key?

[waxlamp]

@rcpeene, I have run into this myself, and because I don't really use my system keyring, I end up doing something like:

$ export DANDI_API_KEY=<paste in API key here>

and then the CLI will no longer prompt for passwords etc. Just be careful about who can see your screen etc., as you want to protect that key from unauthorized use by others.

jwodder may have a more comprehensive solution but exporting your key to your shell should get you unstuck.

[rcpeene]

@jwodder I apologize, I left this out of the snippet

client = dandiapi.DandiAPIClient(token=dandi_api_key)
dandiset = client.get_dandiset(dandiset_id)

Here is the keyring prompt:

[jwodder]

@rcpeene The download() function uses its own client instance, so doing client = dandiapi.DandiAPIClient(token=dandi_api_key) has no effect on it. As @waxlamp said, setting the API key via an environment variable may be the simplest way to pass it to download().

It seems you previously stored your API key in an encrypted keyring file on the system. You can remove this keyfile and start over by deleting ~/.local/share/python_keyring/crypted_pass.cfg. See the handbook for more information on how dandi uses keyrings.

[rcpeene]

This makes sense. On my local machine I already have the DANDI_API_KEY defined which is probably why I didn't have the problem.

I still don't understand why entering my dandi api key into the keyring prompt returns 'incorrect password' though

[jwodder]

I still don't understand why entering my dandi api key into the keyring prompt returns 'incorrect password' though

If you mean the "Please enter password for encrypted keyring" prompt, that's because it's not asking you for your API key. Encrypted keyfiles are well, encrypted, and that encryption involves a user-defined password that would have been set when you first created the keyfile, so now it's asking you for the password used to encrypt the keyfile so it can decrypt the file and retrieve the API key stored within.

[rcpeene]

I've encountered this problem again. On dandihub, I have set DANDI_API_KEY as an environment variable and deleted the file at ~/.local/share/python_keyring/crypted_pass.cfg. Still getting the keyring prompt.

[rcpeene]

When I try on google colab, I instead get the following prompts:

Please provide API Key for dandi: ** Please set a password for your new keyring:

Seems like keyring is being setup automatically?

[kabilar]

Hi @jwodder, I am able to reproduce this issue for an embargoed Dandiset on JupyterHub. I am not sure why I am being prompted for the API key since it is declared in the DANDI_API_KEY environment variable. It only occurs when running the following commands in a Jupyter Notebook on DandiHub, but not when using the terminal in DandiHub. Any suggestions would be appreciated. Thank you.

Steps to reproduce

1. I removed the keyfile at ~/.local/share/python_keyring/crypted_pass.cfg.

2. In a Jupyter notebook on DandiHub, using the CLI:

   Commands

   !export DANDI_API_KEY=<dandi_api_key>
   !dandi download https://api.dandiarchive.org/api/assets/9a4b64df-433c-4efe-a1e7-2ae55b3f95ac/download/

   Returns

   Please provide API Key for dandi: 

3. In a Jupyter notebook on DandiHub, using the Python API: Commands

   !export DANDI_API_KEY=<dandi_api_key>

   from dandi.download import download
   download('https://api.dandiarchive.org/api/assets/9a4b64df-433c-4efe-a1e7-2ae55b3f95ac/download/','.')

   Returns

   Please provide API Key for dandi:  <dandi_api_key>
   Please set a password for your new keyring:  ········
   Please confirm the password:

[yarikoptic]

Just to dismiss some confusion, @kabilar you are running

!export DANDI_API_KEY=<dandi_api_key>
!dandi download https://api.dandiarchive.org/api/assets/9a4b64df-433c-4efe-a1e7-2ae55b3f95ac/download/

in two different cells? then I guess sessions are not shared etc, so effect of export would be lost for the next command. Try smth like

!DANDI_API_KEY=<dandi_api_key> dandi download https://api.dandiarchive.org/api/assets/9a4b64df-433c-4efe-a1e7-2ae55b3f95ac/download/

alternatively, you could just set it in env within that ipython notebook:

import os
os.environ['DANDI_API_KEY'] = "yourkey"

before you import dandi etc so it could be picked up by that process and its children (may be... ;))

Similarly the export you do in a child subshell in 3. has no effect on the state of python process there. If within ipython notebook and just to do it interactively, can simply do how we do it in https://github.com/dandi/example-notebooks/blob/master/dandi/archive_stats.ipynb

client = DandiAPIClient()
client.dandi_authenticate()

which should prompt you for key if it is unknown.