Closed mzfr closed 3 years ago
Changes Missing Coverage | Covered Lines | Changed/Added Lines | % | ||
---|---|---|---|---|---|
tanner/sessions/session_analyzer.py | 2 | 3 | 66.67% | ||
tanner/migration_script.py | 8 | 44 | 18.18% | ||
<!-- | Total: | 10 | 47 | 21.28% | --> |
Files with Coverage Reduction | New Missed Lines | % | ||
---|---|---|---|---|
tanner/utils/asyncmock.py | 2 | 85.71% | ||
tanner/emulators/rfi.py | 21 | 72.15% | ||
<!-- | Total: | 23 | --> |
Totals | |
---|---|
Change from base Build 1392: | -1.9% |
Covered Lines: | 1570 |
Relevant Lines: | 2066 |
while testing, I got
psycopg2.errors.NotNullViolation: null value in column "user_agent" violates not-null constraint
DETAIL: Failing row contains (abd0fc36-d96d-4cba-8771-b5417bcc6cb1, 410c054a-3029-4e27-8172-c8679d75634a, 172.19.0.1, 50002, null, null, null, 0, null, 2019-10-12 02:52:00, 2019-10-12 02:52:00, 85598.04081632652, 0, 1, 0, 0, null).
You should validate the data before moving it into the postgres
please consider checking all non-null values
psycopg2.errors.NotNullViolation: null value in column "user_agent" violates not-null constraint
DETAIL: Failing row contains (50e5ce97-16d4-41be-8e61-ddaeb99d0ab6, 77ce8271-cd9e-4cc1-9cd3-b5b93098a157, 192.168.16.1, 36872, null, null, null, 0, null, 2019-07-22 07:32:24, 2019-07-22 07:32:24, 107546.2564102564, 0, 1, 0, 0, null).
Don't forget to set values according to their types too :)
psycopg2.errors.InvalidTextRepresentation: invalid input syntax for type integer: "N/A"
LINE 1: ...58', 2.8084954716092154, 0.3560633938033859, 231, 'N/A', 'N/... ```
@mzfr the problem not only in integer, but in all the types with missing values
psycopg2.errors.InvalidTextRepresentation: invalid input syntax for type double precision: "N/A"
LINE 1: ...47:46', '2019-07-26 07:47:46', 89240.51063829787, 'N/A', 1, ...
Is it because of the Float and Integer?
In our system, zip
code is Integer, but I found out that it's not correct, e.g. in Japan, zip codes contain -
and some US states have 0 at the beginning of the ZIP code, which is not a valid integer. So I'd recommend changing the type to string
@afeena should I update the code as well as the schema for the postgres?
@mzfr yes, please go ahead :)
@mzfr I tested it and found some problems of migration, to make it easier, I will merge this PR and upload the changes as a separate commit :)
This script can help people in migrating their data from old redis setup to new postgres model.