ZSTD compression is not supported

[eriviere-b]

Currently parquet-viewer does not support ZSTD compression which is a standard compression method.

{"error":"while reading /tmp/SB17-er-ursf_gen-crawls-list.parquet: Error: invalid compression method: ZSTD"}

[dvirtz]

Thanks for opening the issue. What backend do you use (parquet-viewer.backend in the settings)?

[eriviere-b]

I am using the Parquets backend.

[uditrana]

Ran into this issue today as well 👍🏾.

while reading path/to/file: Error: Failed to open path/to/file: Support for codec 'zstd' not built

Using arrow backend

[dvirtz]

:tada: This issue has been resolved in version 2.4.0 :tada:

The release is available on:

• Visual Studio Marketplace
• GitHub release

Your semantic-release bot :package::rocket:

[dvirtz]

This is now fixed for the arrow backend.

[eriviere-b]

When trying to open the Parquet file with the arrow backend, I receive this error: "Error: cannot find prebuilt arrow module, either build the module or use another backend" I am on macOS with Apple Silicon M1 processor, how am I supposed to build the arrow module ?

[dvirtz]

Sorry about that. There's no free M1 VMs currently available for GitHub actions. You can either try to use the parquet-tools backend or try to build the module as follows:

1. make sure you have node.js, pipenv and a C++ compiler installed
2. checkout the extension sources
3. run npm i
4. run npm run build
5. copy the resulting module folder from packages/parquet-reader/prebuilds to <extension folder>/packages/parquet-reader/prebuilds (you can get the extension folder using the Extensions: Open Extensions Folder command)

[dvirtz]

@eriviere-b the latest release has built-in support for Apple M1. I'd be happy to get your feedback

[eriviere-b]

@dvirtz for each engine:

• parquet-tools: there is a pop-up saying "opening {my filename}. Source: parquet-viewer (Extension)", and a loading bar but nothing happens afterward.
• parquet: I still have this error: {"error":"while reading /tmp/SB17-er-ursf_gen-crawls-list.parquet: Error: invalid compression method: ZSTD"}
• arrow: I do not have the C++ compiler installed and I haven't tried to install it.

[dvirtz]

Thanks for that detailed feedback @eriviere-b I specifically meant the arrow backend. It shouldn't require having a C++ compiler installed now as the module is prebuilt and packaged with the extension in CI thanks to codemagic.io supporting M1.

[eriviere-b]

I still receive the same error: "Error: cannot find prebuilt arrow module, either build the module or use another backend"

[dvirtz]

Just to make sure you're on version v2.4.1, right?

[eriviere-b]

Yes, I am loading the latest version of the extension every time I try.

[dvirtz]

Thanks. Can you please tell me what is printed when you run

node -e 'const os = require(\"os\"); console.log(`${os.platform()}-${os.arch()}`)'

[eriviere-b]

darwin-arm64:

[dvirtz]

That's what expected. I released a new version v2.4.2 with some more logging if you don't mind trying. Also if you can turn on the logging to panel option (parquet-viewer.logging.panel) and paste the content of the parquet-viewer output window here. Thanks for your patient.

[melaanya]

experienced the same problem today as @eriviere-b being on 2.4.2

[dvirtz]

I managed to reproduce this on a friend's M1 machine. The error is:

dlopen(/Users/mgunda@roku.com/.vscode/extensions/dvirtz.parquet-viewer-2.4.2/node_modules/parquet-reader/prebuilds/arrow-parquet-reader-darwin-arm64/node-napi-v6.node, 0x0001): tried: '/Users/mgunda@roku.com/.vscode/extensions/dvirtz.parquet-viewer-2.4.2/node_modules/parquet-reader/prebuilds/arrow-parquet-reader-darwin-arm64/node-napi-v6.node' (not a mach-o file)

Not sure how it works on the CI machine.

[dvirtz]

M1 should be fixed with v2.4.3.

[melaanya]

I have an M1, and on v2.4.3 currently (just updated and reloaded) and the issue persists:

{"error":"while reading /Users/annaberger/Downloads/data.parquet: Error: cannot find prebuilt arrow module, either build the module or use another backend: Error: dlopen(/Users/annaberger/.vscode/extensions/dvirtz.parquet-viewer-2.4.3/node_modules/parquet-reader/prebuilds/arrow-parquet-reader-darwin-arm64/node-napi-v6.node, 0x0001): tried: '/Users/annaberger/.vscode/extensions/dvirtz.parquet-viewer-2.4.3/node_modules/parquet-reader/prebuilds/arrow-parquet-reader-darwin-arm64/node-napi-v6.node' (not a mach-o file), '/System/Volumes/Preboot/Cryptexes/OS/Users/annaberger/.vscode/extensions/dvirtz.parquet-viewer-2.4.3/node_modules/parquet-reader/prebuilds/arrow-parquet-reader-darwin-arm64/node-napi-v6.node' (no such file), '/Users/annaberger/.vscode/extensions/dvirtz.parquet-viewer-2.4.3/node_modules/parquet-reader/prebuilds/arrow-parquet-reader-darwin-arm64/node-napi-v6.node' (not a mach-o file)"}

[dvirtz]

Sorry for that, the fix was only integrated in v2.4.4.

[eriviere-b]

Now it works with the arrow engine. Thanks @dvirtz !