No magic number

[alphara]

Hello,

I decompress bz2 file with help of unbzip2-stream in this way:

request.get("http://data.phishtank.com/data/online-valid.json.bz2") .pipe(unbzip2Stream()) .pipe(fs.createWriteStream(__dirname + '/../tmp/online-valid.json'));

It gives me error: No magic number found

But online-valid.json.bz2 is valid bz2 archive.

[regular]

Hi alphara, it is neither a problem with the file nor the decompressor. I ran it through this test and it decompresses just fine. So I guess it is an encoding issue in the stream you get from request. It is probably not a binary stream.

If you are using request beware that it returns an UTF-8 stream by default. From the docs

encoding - Encoding to be used on setEncoding of response data. If null, the body is returned as a Buffer. Anything else (including the default value of undefined) will be passed as the encoding parameter to toString() (meaning this is effectively utf8 by default). (Note: if you expect binary data, you should set encoding: null.)

So, try something like:

request({url: "http://data.phishtank.com/data/online-valid.json.bz2", encoding: null})
    .pipe(unbzip2Stream())
    .pipe(fs.createWriteStream(__dirname + '/../tmp/online-valid.json'));

that should work.

[alphara]

regular,

I tried request with parameter "encoding: null" as you say, and saw that unbzip2Stream() works fine!

Thanks to you we will reduce incoming traffic to 1.3 GB / month)

[regular]

nice. curious: traffic to what website?

[alphara]

Our website is trafficdefender.net . Currently in beta.