search

clrnd / discogs2pg

Discogs to PostgreSQL importer
BSD 3-Clause "New" or "Revised" License
35 stars 8 forks source link

failed to parse command status #3

Closed Paulie87 closed 7 years ago

Paulie87 commented 7 years ago

When trying to parse releases i get the following error every time.

$ stack exec -- discogs2pg -c "host=localhost dbname=discogs_data" discogs_20161001_releases.xml Parsing releases... release = 7642045 release_artist = 8870676 release_extraartist = 26472485 release_label = 8651410 discogs2pg: user error (Database.PostgreSQL.Simple.Copy.putCopyEnd: failed to parse command status)

The following tables do not get populated. release_company release_format release_identifier release_video track track_artist track_extraartist

clrnd commented 7 years ago

Thanks for reporting.

This was a fun one. The problem is this release, One Vigintillion(10^63), which supposedly has "1000000000000000000000000000000000000000000000000000000000000001" formats, which is not in the integer range of Postgres.

I pushed a patch that skips this release (there is a list of avoided items). Or you could just change release_format.qty from integer to text.

I don't know if there are other cases like this, since I don't have enough hard disk space to load the entire dump. You could always use text for everything and handle the data after loading with Postgres which is quite fast.

Cheers, Eze

Paulie87 commented 7 years ago

I went for the text option to avoid issues in the future. I must say discogs2pg is blazingly fast compared to other solutions. Well done.