========= twarc2sql

.. image:: https://img.shields.io/pypi/v/twarc2sql.svg :target: https://pypi.python.org/pypi/twarc2sql

.. image:: https://readthedocs.org/projects/twarc2sql/badge/?version=latest :target: https://twarc2sql.readthedocs.io/en/latest/?version=latest :alt: Documentation Status

.. image:: https://github.com/unna97/twarc2sql/actions/workflows/test.yaml/badge.svg :target: https://github.com/unna97/twarc2sql/actions/workflows/test.yaml :alt: Tests

.. image:: https://static.pepy.tech/personalized-badge/twarc2sql?period=total&units=international_system&left_color=black&right_color=orange&left_text=Total-Downloads :target: https://pepy.tech/project/twarc2sql

.. image:: https://static.pepy.tech/personalized-badge/twarc2sql?period=week&units=international_system&left_color=black&right_color=red&left_text=Week-Downloads :target: https://pepy.tech/project/twarc2sql

.. image:: https://static.pepy.tech/personalized-badge/twarc2sql?period=month&units=international_system&left_color=black&right_color=yellowgreen&left_text=Month-Downloads :target: https://pepy.tech/project/twarc2sql

This package converts jsonl file generated by twarc2 to sql database in an opnionated way.

Free software: MIT license
Documentation: https://twarc2sql.readthedocs.io.

Features

This package converts jsonl file generated by twarc2 to a postgres sql database in an opnionated way.
It creates a database with multiple tables that can be found in the documentation & models.py file.

Installation

You can install twarc2sql using pip:

.. code-block:: console

$ pip install twarc2sql

Usage

.. code-block:: python

    import twarc2sql

    twarc2sql.connect_to_db_and_upload(
        "folderpath/to/jsonl/file",
        "jsonl_file",
        "twarc_task_type",
        "env_file_with_db_information",
    )

Example of env file:

.. code-block:: console

DB_NAME=postgres
DB_USER=postgres
DB_PASSWORD=postgres
DB_HOST=localhost
DB_PORT=5432

Credits

This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.

.. Cookiecutter: https://github.com/audreyr/cookiecutter .. audreyr/cookiecutter-pypackage: https://github.com/audreyr/cookiecutter-pypackage

unna97 / twarc2sql

readme