kaue / jsonexport

{} → :page_facing_up: it's easy to convert JSON to CSV
http://kaue.github.io/jsonexport/
Apache License 2.0
247 stars 41 forks source link

speed question/issue #75

Open m-prof opened 4 years ago

m-prof commented 4 years ago

This is a follow-up to: https://github.com/kaue/jsonexport/issues/64

Thanks so much kaue! This works now for me, for all files I checked.

I did have a follow-up on speed: things slow down for files 1.5M or higher. For me (Win 10 Pro, Surface Pro, 16G RAM, Intel i7-7660U) when I tried a 3.9M file it took 42 minutes (command: jsonexport <json_filename> <output_filename>. Is this sort of time to complete expected?

kaue commented 4 years ago

@m-prof can you try running the benchmark/benchmark.js?

m-prof commented 4 years ago

Sorry for being a noob but I am not getting this to work: I am trying ~\jsonexport\benchmark>node benchmark.js from a command prompt and I get error messages:

internal/modules/cjs/loader.js:976
  throw err;
  ^

Error: Cannot find module 'benchmark'
Require stack:
- \npm\node_modules\jsonexport-master\benchmark\benchmark.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:973:15)
    at Function.Module._load (internal/modules/cjs/loader.js:855:27)
    at Module.require (internal/modules/cjs/loader.js:1033:19)
    at require (internal/modules/cjs/helpers.js:72:18)
    at Object.<anonymous> (\npm\node_modules\jsonexport-master\benchmark\benchmark.js:3:17)
    at Module._compile (internal/modules/cjs/loader.js:1144:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1164:10)
    at Module.load (internal/modules/cjs/loader.js:993:32)
    at Function.Module._load (internal/modules/cjs/loader.js:892:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '\\npm\\node_modules\\jsonexport-master\\benchmark\\benchmark.js'
  ]
}

I assume I am not executing this correctly!

kaue commented 4 years ago

@m-prof you are almost there :) try running npm install first

m-prof commented 4 years ago

Thanks. Not sure if this is sufficient but here is the output (there was a bunch of error messages also as well, let me know if I should post):

npm\node_modules\jsonexport\benchmark>node benchmark.js
Executed benchmark against node module: "jsonexport"
Count (1), Cycles (1), Elapsed (512.53 sec), Hz (0.00975936285335975 ops/sec)

Executed benchmark against node module: "jsonexport-stream"
Count (1), Cycles (1), Elapsed (10.388 sec), Hz (0.8795467729800702 ops/sec)

PS There were two output files (out.csv and out-no-streams.csv), the converted version of data.json. I can post them though they look fine!