search

rheinwerk-verlag / pganonymize

A commandline tool for anonymizing PostgreSQL databases
http://pganonymize.readthedocs.io/
Other
42 stars 26 forks source link

Script fails when a table column as upper case characters #16

Closed rap2hpoutre closed 3 years ago

rap2hpoutre commented 3 years ago

Script fails when a table column as upper case characters

Traceback (most recent call last):
  File "/usr/local/bin/pganonymize", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/pganonymizer/__main__.py", line 10, in main
    main()
  File "/usr/local/lib/python3.9/site-packages/pganonymizer/cli.py", line 71, in main
    anonymize_tables(connection, schema.get('tables', []), verbose=args.verbose)
  File "/usr/local/lib/python3.9/site-packages/pganonymizer/utils.py", line 39, in anonymize_tables
    import_data(connection, column_dict, table_name, table_columns, primary_key, data)
  File "/usr/local/lib/python3.9/site-packages/pganonymizer/utils.py", line 140, in import_data
    copy_from(connection, data, 'source', table_columns)
  File "/usr/local/lib/python3.9/site-packages/pganonymizer/utils.py", line 119, in copy_from
    cursor.copy_from(new_data, table, sep=COPY_DB_DELIMITER, null='\\N', columns=columns)
psycopg2.errors.UndefinedColumn: column "createdat" of relation "source" does not exist
hkage commented 3 years ago

Hi, that means the column name in your database uses upper or mixed cased characters and you specified a lowercase column name in your YAML scheme? Or does this error occurs, even if you are using the same case in your scheme?

bobslee commented 3 years ago

Any info on this one? @rap2hpoutre

rap2hpoutre commented 3 years ago

@bobslee Oops, no sorry about that. I ended creating my own pganonymizer: https://github.com/rap2hpoutre/pg-anonymizer 😅

hkage commented 3 years ago

Sorry to hear that you can't use our project for your use case. I will try to improve it for those who want to use a Python based (and Python 2 supporting) tool.

We still don't use the anonymizer for our development databases therefore it takes some time and extra effort to improve this project. But I am more than happy if other contributors provide some fixes and enhancements.

rap2hpoutre commented 3 years ago

Thank you for your answer (and sorry for self-promotion, I can remove it!)

hkage commented 3 years ago

I made a fix that preserves the column and table names. That should fix issues with mixed or upper case names. As you described in your project, the anomyizer still cannot handle other PostgreSQL schemas than "public", but I will fix that in another commit.

You don't have to delete the link. But if have the time maybe you could try out the latest release of postgresql-anonymizer (I will upload it soon) and give me a short feedback if the error still occurs on your side.