Invalid buffer issue

[amenon21]

simple.zip Tried applying the same usage from a local ts file (generated using Angular CLI) import { Component } from '@angular/core'; import {readFileSync } from "file-system"; import { Table } from "apache-arrow";

@Component({ selector: 'app-root', templateUrl: './app.component.html', styleUrls: ['./app.component.css'] }) export class AppComponent { title = 'app';

}

const arrow = readFileSync('simple.arrow'); const table = Table.from([arrow]);

console.log(table.toString());

I am getting the below error that is pointing to the arrow.js

Error: Invalid Arrow buffer at readSchema (C:\Users\amen0007\arrowjs\node_modules\apache-arrow\reader\arrow.js:196:15) at C:\Users\amen0007\arrowjs\node_modules\apache-arrow\reader\arrow.js:122:27 at step (C:\Users\amen0007\arrowjs\node_modules\tslib\tslib.js:133:27) at Object.next (C:\Users\amen0007\arrowjs\node_modules\tslib\tslib.js:114:57) at step (C:\Users\amen0007\arrowjs\node_modules\tslib\tslib.js:118:101) at Object.next (C:\Users\amen0007\arrowjs\node_modules\tslib\tslib.js:114:57) at Function.Table.from (C:\Users\amen0007\arrowjs\node_modules\apache-arrow\vector\table.js:33:87) at C:\Users\amen0007\arrowjs\test.js:12:22 at tryToString (fs.js:456:3) at FSReqWrap.readFileAfterClose [as oncomplete] (fs.js:443:12)

Attached simple.arrow.

Is there something I am missing here, can you please guide?

[trxcllnt]

@amenon21 are you using apache-arrow@0.2.0 on npm, or the 0.3.0 prerelease hosted in this repo?

If you're using apache-arrow@0.2.0 on npm, @angular/cli is probably grabbing the ES5/UMD file, which didn't minify correctly and was published by mistake. That will be fixed in 0.3.0, but in the meantime you can change the "apache-arrow" entry in your package.json to point to this repository. This worked for me:

$ cat package.json 
{
  "dependencies": {
    "apache-arrow": "graphistry/arrow"
  }
}

$ npx arrow2csv -f simple.arrow 
"row_id" | "N_NATIONKEY: Int64" |   "N_NAME: Utf8" | "N_REGIONKEY: Int64" |      "N_COMMENT: Utf8"
       0 |                [0,0] |        "ALGERIA" |                [0,0] | " haggle. carefully f"
       1 |                [1,0] |      "ARGENTINA" |                [1,0] | "al foxes promise sly"
       2 |                [2,0] |         "BRAZIL" |                [1,0] | "y alongside of the p"
       3 |                [3,0] |         "CANADA" |                [1,0] | "eas hang ironic, sil"
       4 |                [4,0] |          "EGYPT" |                [4,0] | "y above the carefull"
       5 |                [5,0] |       "ETHIOPIA" |                [0,0] | "ven packages wake qu"
       6 |                [6,0] |         "FRANCE" |                [3,0] | "refully final reques"
       7 |                [7,0] |        "GERMANY" |                [3,0] | "l platelets. regular"
       8 |                [8,0] |          "INDIA" |                [2,0] | "ss excuses cajole sl"
       9 |                [9,0] |      "INDONESIA" |                [2,0] | " slyly express asymp"
      10 |               [10,0] |           "IRAN" |                [4,0] | "efully alongside of "
      11 |               [11,0] |           "IRAQ" |                [4,0] | "nic deposits boost a"
      12 |               [12,0] |          "JAPAN" |                [2,0] | "ously. final, expres"
      13 |               [13,0] |         "JORDAN" |                [4,0] | "ic deposits are blit"
      14 |               [14,0] |          "KENYA" |                [0,0] | " pending excuses hag"
      15 |               [15,0] |        "MOROCCO" |                [0,0] | "rns. blithely bold c"
      16 |               [16,0] |     "MOZAMBIQUE" |                [0,0] | "s. ironic, unusual a"
      17 |               [17,0] |           "PERU" |                [1,0] | "platelets. blithely "
      18 |               [18,0] |          "CHINA" |                [2,0] | "c dependencies. furi"
      19 |               [19,0] |        "ROMANIA" |                [3,0] | "ular asymptotes are "
      20 |               [20,0] |   "SAUDI ARABIA" |                [4,0] | "ts. silent requests "
      21 |               [21,0] |        "VIETNAM" |                [2,0] | "hely enticingly expr"
      22 |               [22,0] |         "RUSSIA" |                [3,0] | " requests against th"
      23 |               [23,0] | "UNITED KINGDOM" |                [3,0] | "eans boost carefully"
      24 |               [24,0] |  "UNITED STATES" |                [1,0] | "y final packages. sl"

[amenon21]

Thanks @trxcllnt, I was pointing to 0.2.0, I have pointed to 0.3.0 in this repo modified package.json - "apache-arrow": "graphistry/arrow" and did an install. package.lock gets generated as - "apache-arrow": { "version": "github:graphistry/arrow#a506e414df37eecf23a09bd6f97c8a782b46e66c", "requires": { "@types/flatbuffers": "1.6.5", "@types/node": "6.0.97", "@types/text-encoding-utf-8": "1.0.1", "command-line-args": "5.0.1", "command-line-usage": "4.1.0", "flatbuffers": "github:trxcllnt/flatbuffers-esm#0ed890546a45c361665b1778f856b4a48cc22c8a", "json-bignum": "0.0.3", "text-encoding-utf-8": "1.0.2", "tslib": "1.9.0" } }, But getting the below error now - ERROR in node_modules/apache-arrow/data.d.ts(60,22): error TS2415: Class 'BoolData' incorrectly extends base class 'FlatData'. Types of property 'sliceData' are incompatible. Type '(data: Uint8Array) => Uint8Array' is not assignable to type '(data: Uint8Array & TypedArray, offset: number, length: number) => TypedArray'. Type 'Uint8Array' is not assignable to type 'TypedArray'. Types of property 'copyWithin' are incompatible. Type '(target: number, start: number, end?: number) => Uint8Array' is not assignable to type '(target: number, start: number, end?: number) => TypedArray'. Type 'Uint8Array' is not assignable to type 'TypedArray'. node_modules/apache-arrow/type.d.ts(136,47): error TS2344: Type 'TArrayType' does not satisfy the constraint 'TypedArray'. Type 'Uint8Array | Int32Array | Int8Array | Int16Array | Uint16Array | Uint32Array' is not assignable to type 'TypedArray'. Type 'Uint8Array' is not assignable to type 'TypedArray'. Types of property 'copyWithin' are incompatible. Type '(target: number, start: number, end?: number) => Uint8Array' is not assignable to type '(target: number, start: number, end?: number) => TArrayType'. Type 'Uint8Array' is not assignable to type 'TArrayType'. node_modules/apache-arrow/type.d.ts(172,47): error TS2344: Type 'TArrayType' does not satisfy the constraint 'TypedArray'. Type 'FloatArray' is not assignable to type 'TypedArray'. Type 'Float32Array' is not assignable to type 'TypedArray'. Types of property 'copyWithin' are incompatible. Type '(target: number, start: number, end?: number) => Float32Array' is not assignable to type '(target: number, start: number, end?: number) => TArrayType'. Type 'Float32Array' is not assignable to type 'TArrayType'.

[trxcllnt]

@amenon21 Ah let me regenerate this repo from the latest Arrow source. TS 2.7.1 did something to break generics for one of our methods, but we've fixed it. Adding skipLibCheck: true to your tsconfig should clear this in the meantime.

[amenon21]

@trxcllnt thanks, I got the build working. However I get an uncaught type error with accessing readFileSync from app.component.ts ERROR TypeError: fs.readFileSync is not a function at AppComponent.ngOnInit (app.component.ts:16) at checkAndUpdateDirectiveInline (core.js:12352) at checkAndUpdateNodeInline (core.js:13876)

I read that "fs.readFileSync will never truly work on the client as it's Node functionality and the browser is not Node." Is that correct, anything I am missing?

[trxcllnt]

@amenon21 Ok, I've updated this repo with the latest built files from Arrow/master.

I read that "fs.readFileSync will never truly work on the client as it's Node functionality and the browser is not Node."

Yes, the fs module is node-only. In the browser you need to use something like XHR or FileReader to load in binary data from external sources. We have an example HTML file that uses the FileReader in the main Arrow repository here.

[trxcllnt]

@amenon21 the example still needs to be updated to the new API, but here's the diff:

Starting on line 56, it currently says:

  for (let column of arrowTable.columns) {
    addCell(header_row, "th", column.name);
  }

But instead it should be:

  for (let field of arrowTable.schema.fields) {
    addCell(header_row, "th", field.name);
  }

[amenon21]

Thanks @trxcllnt