c-3lab / dim

📦 dim: Manage the open data in your project like a package manager.
MIT License
126 stars 21 forks source link

Python library to simplify the interaction with dim #82

Open lisphilar opened 2 years ago

lisphilar commented 2 years ago

Thank you for creating this project :) Data installation manager is absolutely required for open source community. I faced some difficulties when developing a dataset and a data analysis tool with Python regarding COVID-19.

Is it possible to add Python (+R?) library to simplify the interaction with dim? (I'm not sure we can call Deno from Python...) Users may use the new library as follows.

  1. Install the library, like poetry add dim-python
  2. Write settings on "pyproject.toml" with commands. This TOML format file is the standard Python library management file currently.
[tool.dim]
directory = './data_files'
datasets = [
    {
        name = 'example',
        url = 'https://example.com',
        unzip = true,
        forced = true,
        encoding = 'utf-8',
        postprocess = ["poetry run python ./tests/test_custom_command.py",],
    },
]
  1. Update datasets with poetry run dim update, or

  2. Update/load the dataset with Python scripts.

    import dim
    dim.config(settings='./data_files/dim-lock.json')
    data = dim.load(name='example')

I'm just a new user, but very interested in this project.

ryo-ma commented 2 years ago

Thank you for your interest. I am planning to support the execution of dim from the library.

If you are interested, would you like to develop dim together? Just getting your opinion is fine. I want to grow this open source together!

lisphilar commented 2 years ago

Many thanks for your positive response! I will be happy to join this project as a developer.

ryo-ma commented 2 years ago

I am trying to develop dim as an open community. I already have Slack channel, so please join if you can participate. I would like to communicate on Slack.

https://join.slack.com/t/c3lab-hq/shared_invite/zt-v6zz66n9-1VYkVXC4zoQViWSMdzMTLg

lisphilar commented 2 years ago

@ryo-ma Thank you for the invitation for Slack. I successfully join to the Slack team. This issue is still open to record the status of the library development.

ryo-ma commented 2 years ago

I have created a repository. https://github.com/c-3lab/dim-python