rheinwerk-verlag / pganonymize

A commandline tool for anonymizing PostgreSQL databases
http://pganonymize.readthedocs.io/
Other
41 stars 26 forks source link

Add json support #28

Closed nurikk closed 3 years ago

nurikk commented 3 years ago

This commit adds native support for jsonb columns, example:

  - billable_transactions:
      primary_key: id
      chunk_size: 500000
      fields:
        - metadata.initiator_name:
            provider:
              name: fake.name
> \d billable_transactions
                                                Table "public.billable_transactions"
          Column           |            Type             | Collation | Nullable |                      Default
---------------------------+-----------------------------+-----------+----------+---------------------------------------------------
 id                        | bigint                      |           | not null | nextval('billable_transactions_id_seq'::regclass)
 metadata                  | jsonb                       |           |          | '{}'::jsonb

Additionally it adds support for

---------- coverage: platform darwin, python 3.9.2-final-0 -----------
Name                         Stmts   Miss  Cover   Missing
----------------------------------------------------------
pganonymizer/__init__.py         2      0   100%
pganonymizer/__main__.py        11     11     0%   2-16
pganonymizer/cli.py             54      0   100%
pganonymizer/constants.py        4      0   100%
pganonymizer/exceptions.py       5      0   100%
pganonymizer/providers.py       77      0   100%
pganonymizer/utils.py          165      0   100%
pganonymizer/version.py          1      0   100%
----------------------------------------------------------
TOTAL                          319     11    97%
nurikk commented 3 years ago
---------- coverage: platform darwin, python 3.9.2-final-0 -----------
Name                         Stmts   Miss  Cover   Missing
----------------------------------------------------------
pganonymizer/__init__.py         2      0   100%
pganonymizer/__main__.py        13     13     0%   2-20
pganonymizer/cli.py             58      8    86%   31-33, 65-66, 90, 94-95
pganonymizer/constants.py        5      0   100%
pganonymizer/exceptions.py       5      0   100%
pganonymizer/providers.py       73      2    97%   33, 60
pganonymizer/utils.py          185     15    92%   51, 58, 81, 140-144, 161-162, 280, 312-318
pganonymizer/version.py          1      0   100%
----------------------------------------------------------
TOTAL                          342     38    89%

huh, not bad :)

hkage commented 3 years ago
---------- coverage: platform darwin, python 3.9.2-final-0 -----------
Name                         Stmts   Miss  Cover   Missing
----------------------------------------------------------
pganonymizer/__init__.py         2      0   100%
pganonymizer/__main__.py        13     13     0%   2-20
pganonymizer/cli.py             58      8    86%   31-33, 65-66, 90, 94-95
pganonymizer/constants.py        5      0   100%
pganonymizer/exceptions.py       5      0   100%
pganonymizer/providers.py       73      2    97%   33, 60
pganonymizer/utils.py          185     15    92%   51, 58, 81, 140-144, 161-162, 280, 312-318
pganonymizer/version.py          1      0   100%
----------------------------------------------------------
TOTAL                          342     38    89%

huh, not bad :)

Now we`re getting to a serious test coverage :-)

nurikk commented 3 years ago

Hi, @hkage , kindly help me with review and testing :) this PR is ready for review

hkage commented 3 years ago

Hi, @hkage , kindly help me with review and testing :) this PR is ready for review

Hey @nurikk , thanks for the PR :-) I will take a look into it this week.

nurikk commented 3 years ago

Hi @nurikk , thanks again for this huge PR and a bunch of great new features and additions to this project. This PR alone should result in a version bump to 1.0 instead of 0.6 😄

There are only some small annotations, everything else works perfectly for me, also on my bigger test database.

I would add some more Sphinx documentation after the annotations were fixed and the PR merged, but you don't have to add it right here within the PR.

I've updated PR

hkage commented 3 years ago

Hi @nurikk , thanks again for this huge PR and a bunch of great new features and additions to this project. This PR alone should result in a version bump to 1.0 instead of 0.6 smile There are only some small annotations, everything else works perfectly for me, also on my bigger test database. I would add some more Sphinx documentation after the annotations were fixed and the PR merged, but you don't have to add it right here within the PR.

I've updated PR

LGTM :+1: