pg_dump command failed

[meswanb]

@rap2hpoutre

Even though pg_dump is installed, I get this error when running pg-anonymizer:

npx pg-anonymizer postgres://....
npx: installed 80 in 4.721s
Launching pg_dump
pg_dump command failed. Are you sure it is installed?

root@ip-10-0-00-000:/ # pg_dump --version
pg_dump (PostgreSQL) 12.7 (Ubuntu 12.7-1.pgdg20.04+1)

Anything I can try?

[arnaudambro]

I thought I had the same error, but when you check the source code, the Are you sure it is installed is a basic error message. If you have pg_dump installed, the script is not working because of another error. For instance, my error was that my connection string was wrong.

[itslenny]

@arnaudambro is there a way to see what the error is? Mine throws this error, but does not provide ANY additional details.

Update. I found the line that logs that error and just did throw e so I could see the actual error and got the following

Launching pg_dump
    RangeError: stdout maxBuffer length exceeded
    Code: ERR_CHILD_PROCESS_STDIO_MAXBUFFER

I increased the buffer little by little using the -m flag, but eventually just ran out of memory in general. It seems this library tries to hold the entire DB in memory which obviously won't work for any real production data set so it's more of a toy than anything.

Hope this helps save any future person hoping this library may be a solution from wasting time.

(no offense meant to the author, this is a really cool idea, and well designed, but it can't handle large databases)

[imreACTmd]

I have a little time to try to redo it to do stream processing. I will let you folks know if I could figure it out, but I'm also looking for suggestions to approach this.

[imreACTmd]

All right I managed to redo the code as streams, the working code is in https://github.com/rap2hpoutre/pg-anonymizer/pull/12

[imreACTmd]

This has been resolved.

[arnaudambro]

hello @imreACTmd I still have the problem of @itslenny

RangeError [ERR_CHILD_PROCESS_STDIO_MAXBUFFER]: stdout maxBuffer length exceeded
    at new NodeError (node:internal/errors:371:5)
    at Socket.onChildStdout (node:child_process:459:14)
    at Socket.emit (node:events:390:28)
    at addChunk (node:internal/streams/readable:315:12)
    at readableAddChunk (node:internal/streams/readable:285:11)
    at Socket.Readable.push (node:internal/streams/readable:228:10)
    at Pipe.onStreamRead (node:internal/stream_base_commons:199:23) {
  code: 'ERR_CHILD_PROCESS_STDIO_MAXBUFFER',

I see you're talking about the -m flag, but I don't know how to use it ?

[arnaudambro]

alright, I tried to add -m=5120 to the end of my command and it worked !

[imreACTmd]

just a question, if you run npx pg-anonymizer -v what result do you get?

[arnaudambro]

pg-anonymizer/0.3.0 darwin-x64 node-v16.13.0

[imreACTmd]

weird, we are at 1.0.0:

$ npx pg-anonymizer -v pg-anonymizer/1.0.0 darwin-x64 node-v14.18.2

[arnaudambro]

ah ! I thought that npx purpose was related to having the latest version...

I ran npx pg-anonymizer@latest -v and I have pg-anonymizer/0.6.0 darwin-x64 node-v16.13.0

[imreACTmd]

apparently in version 7 npx started caching downloaded packages.