skale-me / node-parquet

NodeJS module to access apache parquet format files
Apache License 2.0
57 stars 11 forks source link

Hangs on error conditions #54

Open mars opened 6 years ago

mars commented 6 years ago

First of all, thank you for creating a Node module that performs this very special task and for sharing it with us!

I've found two different conditions that cause the host program to hang indefinitely (in Node 8.9.0), never reporting an error or any hint of what the issue could be. I debugged these problems through a couple of arduous debug sessions:

  1. Writing data arrays that contain null/undefined instead of empty Array indexes for optional fields. What is an empty Array index? How does one create an empty Array index? The answer is that it is impossible to explicitly create an empty index. Instead, you have to create an empty Array and then set only the indexes that should contain a value. The remaining indexes are "empty", a little-known aspect of JavaScript Arrays.
  2. Writing a parquet file to a directory that does not exist.

Each of these are not necessarily problems, especially if they're documented. The problem is that this module does not throw errors, instead it hangs.

mars commented 6 years ago

Following-up on my 1. above,

Remy Sharp just published a fantastic post about this topic:
👓 An Adventure in Sparse Arrays