treeverse / lakeFS

lakeFS - Data version control for your data lake | Git for data
https://docs.lakefs.io
Apache License 2.0
4.35k stars 344 forks source link

[Bug]: Can't pass `read_only` to the Python wrapper's repository creation API #7848

Open yonipeleg33 opened 3 months ago

yonipeleg33 commented 3 months ago

What happened?

The Python wrapper is not familiar with the read_only optional argument that can be passed to the createRepository API (this is the operation id in swagger.yml).

Steps to Reproduce:

  1. Write a Python file:
    
    import lakefs
    from lakefs.client import Client

clt = Client( username="", password="", host="", ) repo = lakefs.Repository(repository_id="", client=clt).create( storage_namespace="", read_only=True, )

2. Run it: `python <filename>.py`
3. Expect the following result:

... raise ApiTypeError( lakefs_sdk.exceptions.ApiTypeError: Got an unexpected keyword argument 'read_only' to method create_repository


### Expected behavior

This should work.
For example, the following equivalent code works using the Rust client:
```rust
#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let config = Configuration {
        base_path: "<url>".to_string(),
        basic_auth: Some((
            "<user>".to_string(),
            Some("<password>".to_string()),
        )),
        ..Configuration::default()
    };

    lakefs_sdk::apis::repositories_api::create_repository(
        &config,
        RepositoryCreation {
            name: "<repo>".to_string(),
            storage_namespace: "<namespace>"
                .to_string(),
            read_only: Some(true),
            ..RepositoryCreation::default()
        },
        None,
    )
    .await
    .context("creating repo")?;
    Ok(())
}

lakeFS version

No response

How lakeFS is installed

No response

Affected clients

No response

Relevant log output

Here's the full error produced by the Python code:

  File "/home/<user>/workspace/playground/lakefs-rs-pg/../pg.py", line 9, in <module>
    repo = lakefs.Repository(repository_id="<repo>", client=clt).create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/<user>/workspace/playground/venv/lib/python3.11/site-packages/lakefs/repository.py", line 65, in create
    repo = self._client.sdk_client.repositories_api.create_repository(repository_creation, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/<user>/workspace/playground/venv/lib/python3.11/site-packages/pydantic/v1/decorator.py", line 40, in wrapper_function
    return vd.call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/<user>/workspace/playground/venv/lib/python3.11/site-packages/pydantic/v1/decorator.py", line 134, in call
    return self.execute(m)
           ^^^^^^^^^^^^^^^
  File "/home/<user>/workspace/playground/venv/lib/python3.11/site-packages/pydantic/v1/decorator.py", line 206, in execute
    return self.raw_function(**d, **var_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/<user>/workspace/playground/venv/lib/python3.11/site-packages/lakefs_sdk/api/repositories_api.py", line 91, in create_repository
    return self.create_repository_with_http_info(repository_creation, bare, **kwargs)  # noqa: E501
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/<user>/workspace/playground/venv/lib/python3.11/site-packages/pydantic/v1/decorator.py", line 40, in wrapper_function
    return vd.call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/<user>/workspace/playground/venv/lib/python3.11/site-packages/pydantic/v1/decorator.py", line 134, in call
    return self.execute(m)
           ^^^^^^^^^^^^^^^
  File "/home/<user>/workspace/playground/venv/lib/python3.11/site-packages/pydantic/v1/decorator.py", line 206, in execute
    return self.raw_function(**d, **var_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/<user>/workspace/playground/venv/lib/python3.11/site-packages/lakefs_sdk/api/repositories_api.py", line 153, in create_repository_with_http_info
    raise ApiTypeError(
lakefs_sdk.exceptions.ApiTypeError: Got an unexpected keyword argument 'read_only' to method create_repository


### Contact details

_No response_
N-o-Z commented 2 weeks ago

@yonipeleg33 This is not a bug, it's a feature request. Do we have any compelling business reason to add it?

yonipeleg33 commented 2 weeks ago

@N-o-Z I was not aware of it being a feature request, and it sounds weird to me, TBH - if you don't want to use the read_only flag, just don't use it, what is the motivation behind requesting a feature to remove an optional argument?

Anyway, I opened the bug because I wanted to use this flag and couldn't, and simply assumed it was a missing implementation, no business-y reason.

N-o-Z commented 2 weeks ago

@N-o-Z I was not aware of it being a feature request, and it sounds weird to me, TBH - if you don't want to use the read_only flag, just don't use it, what is the motivation behind requesting a feature to remove an optional argument?

Anyway, I opened the bug because I wanted to use this flag and couldn't, and simply assumed it was a missing implementation, no business-y reason.

What I meant is that the issue you opened is a feature request and not a bug. There was no requirement to support read_only mode in HL Python SDK (it also didn't exist at the time). The purpose of the HL SDK is to provide a simple and user friendly interface for lakeFS in python, not to support all the available functionality - we have the underlying Python SDK for that.

yonipeleg33 commented 2 weeks ago

Oh. So yeah, you can call that a feature request instead of a bug. I think read_only is worthy enough of being added to the higher level APIs the HL SDK provides. That's why I marked it as a bug. But arguably, it can be thought of as a feature request.

LMK whether I should change the title and description accordingly, I'm fine with either way.

N-o-Z commented 2 weeks ago

read_only is used mostly internally, I can't think of a user scenario where they'd want to create a read only repo. You can change the title if you want, I think the discussion we had here also provide enough relevant context