Authentication method is not parsed when supplied in the Snakefile

[Hugovdberg]

The provider settings define an auth setting, however the string value is never parsed using the parse_auth function, and the request then fails with

Traceback (most recent call last):

  File "c:\Temp\conda-envs\snakemake_storage_plugin_http\Lib\site-packages\snakemake\cli.py", line 2068, in args_to_api
    dag_api.execute_workflow(

  File "c:\Temp\conda-envs\snakemake_storage_plugin_http\Lib\site-packages\snakemake\api.py", line 589, in execute_workflow
    workflow.execute(

  File "c:\Temp\conda-envs\snakemake_storage_plugin_http\Lib\site-packages\snakemake\workflow.py", line 1081, in execute
    self._build_dag()

  File "c:\Temp\conda-envs\snakemake_storage_plugin_http\Lib\site-packages\snakemake\workflow.py", line 1037, in _build_dag
    async_run(self.dag.init())

  File "c:\Temp\conda-envs\snakemake_storage_plugin_http\Lib\site-packages\snakemake\common\__init__.py", line 94, in async_run
    return asyncio.run(coroutine)
           ^^^^^^^^^^^^^^^^^^^^^^

  File "c:\Temp\conda-envs\snakemake_storage_plugin_http\Lib\asyncio\runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^

  File "c:\Temp\conda-envs\snakemake_storage_plugin_http\Lib\asyncio\runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "c:\Temp\conda-envs\snakemake_storage_plugin_http\Lib\asyncio\base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^

  File "c:\Temp\conda-envs\snakemake_storage_plugin_http\Lib\site-packages\snakemake\dag.py", line 178, in init
    job = await self.update([job], progress=progress, create_inventory=True)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "c:\Temp\conda-envs\snakemake_storage_plugin_http\Lib\site-packages\snakemake\dag.py", line 940, in update
    await self.update_(

  File "c:\Temp\conda-envs\snakemake_storage_plugin_http\Lib\site-packages\snakemake\dag.py", line 1037, in update_
    await res.file.inventory()

  File "c:\Temp\conda-envs\snakemake_storage_plugin_http\Lib\site-packages\snakemake\io.py", line 265, in inventory
    await asyncio.gather(*tasks)

  File "c:\Temp\conda-envs\snakemake_storage_plugin_http\Lib\site-packages\snakemake_storage_plugin_http\__init__.py", line 159, in inventory
    with self.httpr(verb="HEAD") as httpr:

  File "c:\Temp\conda-envs\snakemake_storage_plugin_http\Lib\contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^

  File "c:\Temp\conda-envs\snakemake_storage_plugin_http\Lib\site-packages\snakemake_storage_plugin_http\__init__.py", line 218, in httpr
    r = request(
        ^^^^^^^^

  File "c:\Temp\conda-envs\snakemake_storage_plugin_http\Lib\site-packages\requests\api.py", line 100, in head
    return request("head", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "c:\Temp\conda-envs\snakemake_storage_plugin_http\Lib\site-packages\requests\api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "c:\Temp\conda-envs\snakemake_storage_plugin_http\Lib\site-packages\requests\sessions.py", line 575, in request
    prep = self.prepare_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^

  File "c:\Temp\conda-envs\snakemake_storage_plugin_http\Lib\site-packages\requests\sessions.py", line 486, in prepare_request
    p.prepare(

  File "c:\Temp\conda-envs\snakemake_storage_plugin_http\Lib\site-packages\requests\models.py", line 372, in prepare
    self.prepare_auth(auth, url)

  File "c:\Temp\conda-envs\snakemake_storage_plugin_http\Lib\site-packages\requests\models.py", line 603, in prepare_auth
    r = auth(self)
        ^^^^^^^^^^

TypeError: 'str' object is not callable

As far as I can see the parse function is only referenced in the metadata on the settings field, but never used. I still find the new plugin structure quite unclearly documented, so I'm not sure whether this metadata field was meant to be automagically used on instantiation, but currently the authentication is broken.

[Hugovdberg]

Another weird thing (but that might be caused by the way snakemake initialises these objects) is that calling parse_auth in StorageProviderSettings.__post_init__ shows that at that stage StorageProviderSettings.auth is always None, but in StorageProvider.__post_init__ the value of self.settings.auth is the not-yet-parsed string value from the snakefile.

[Hugovdberg]

I think I found a piece of the puzzle, auth is a string value that could be tagged, but for some reason when the settings are instantiated the value for all fields is dataclasses.MISSING and therefore not parsed, according to this piece of code: https://github.com/snakemake/snakemake-interface-common/blob/00bc4cafddd0b18a237ff43807202075c2562f78/snakemake_interface_common/plugin_registry/plugin.py#L257-L264

Also, if it would be parsed with the field.metadata.parse_func, the extract_values function would raise an InvalidPluginException on https://github.com/snakemake/snakemake-interface-common/blob/00bc4cafddd0b18a237ff43807202075c2562f78/snakemake_interface_common/plugin_registry/plugin.py#L239C35-L239C57 because the auth metadata only specify a parse_func but not an unparse_func (https://github.com/snakemake/snakemake-storage-plugin-http/blob/1446dd486ce16da2219ba937c39e5c0339142bee/snakemake_storage_plugin_http/__init__.py#L70-L77)

[Hugovdberg]

Documenting my understanding of the problem here, sorry for the spamming of the thread.

As far as I can see the snakemake_storage_plugin_http.StorageProviderSettings are only instantiated using the command line arguments, but not the settings from the Snakefile. That is why the values of all the fields are MISSING, and indeed, specifying the auth method on the command line raises the InvalidPluginException because the unparse_func is not defined.

Some debugging with a modified __setattr__ on the SettingsBase shows that the values are initially set to the default values from the __init__ method, but then once the settings are passed to the StorageProvider the values have been changed without SettingsBase.__setattr__ being called. It turns out that snakemake itself bluntly updates the class dictionary in snakemake.storage, thereby circumventing any __setattr__ or __post_init__ calls.

This then leads to the conclusion that snakemake itself simply expects you to provide the initialised AuthBase derived object in the Snakefile instead of the string that would be parsed from the command line. So that leaves the fact that the command line version of the auth option is broken due to the missing unparse_func, but the Snakefile version works but is badly documented. It actually implies you pass the string instead of the object itself. I will update the original post and title to reflect the current status.

[Hugovdberg]

Opted to create a new issue instead, because this thread does not make much sense in the context of the command line issue.