search

luizperes / simdjson_nodejs

Node.js bindings for the simdjson project: "Parsing gigabytes of JSON per second"
https://arxiv.org/abs/1902.08318
Apache License 2.0
549 stars 25 forks source link

Detect dynamically AVX2 within npm #3

Closed lemire closed 4 years ago

lemire commented 5 years ago

When can detect AVX2 dynamically within npm without user intervention, see for example...

https://github.com/SalvatorePreviti/roaring-node/blob/master/binding.gyp#L24-L29

luizperes commented 5 years ago

That sounds good! I am still to test these bindings in all three platforms (we got an error with windows https://github.com/luizperes/simdjson_nodejs/issues/1) and will totally add the AVX2 checking when I get there!

Thanks @lemire

luizperes commented 5 years ago

Hi @lemire, I think I fixed this issue. However, because I do not have an AVX2 Linux machine, I would like to ask if you could download the branch issue-6 (https://github.com/luizperes/simdjson_nodejs/tree/issue-6) and test if it is still compiling for Linux.

It was needed because we were having a bug where the bindings were only working for Linux (please see #1 and #6). Oh! I used the lead that you gave me https://github.com/SalvatorePreviti/roaring-node/blob/master/binding.gyp! It helped a lot!

I would appreciate if you could give me an update on this issue. I will merge the code onto master when this issue has been fixed. One more thing, please make sure to run the benchmarks just to double-check if it is recognizing your system as AVX2 (I created an simdjson as an alias to JSON when the avx2 flag checking fails...)

Thanks!

lemire commented 5 years ago

@luizperes I have just emailed to your sfu address instructions for how to access (by ssh/linux) Linux servers with AVX2.

Will that help?

lemire commented 5 years ago

(If you don't get my email, please reach out to me at lemire@gmail.com.)

lemire commented 5 years ago

Idea: it is 10x harder to support a platform you cannot directly access.

luizperes commented 5 years ago

@lemire That helps a lot, I received your email, will work on that and will update you here! Thanks!

luizperes commented 5 years ago

Hi @lemire, it is working properly on Linux as well, thanks for the help. Also, if you kept my user would be good so I can test in the future. I realized that the Nan Google V8, might actually be worse than the previous Napi, so I will need to cherry pick some of the commits before I merge it. Thank you again!

lemire commented 5 years ago

Sure. I also have other Linux machines if you want further testing.

@TkTech : I can give you the same access if you need it.

TkTech commented 5 years ago

@lemire thanks for the offer, I luckily have a large collection of random boxes from my osdev days. Did you see the new POWER9 blackbird? First POWER9 I've seen around ~$1k for PCU+board.

@luizperes There is also the Intel Software Development Emulator for win/mac/linux hosts. Will be slow as the dickens, but if you're just doing development and testing it's perfect for emulating AVX2/AVX-512 for a single process.

lemire commented 5 years ago

Anyhow, I am happy to provide access to our machines for this type of research.

Evidently, one can also get access to fun machines on cloud providers like Packet.

(Yes. I have a POWER9 machine. Was way more expensive than that... but now that I have it...)

luizperes commented 5 years ago

Thanks for the info about the emulator @TkTech !

luizperes commented 5 years ago

Hi @lemire, I am detecting AVX2 now on the (merged) branch https://github.com/luizperes/simdjson_nodejs/tree/proper-fix-issue-6. I've tested so far on Mac OS and Linux and it looks good. I am waiting for @davaloper to update me on Windows.

Other than that, please let me know if this issue can be closed. Cheers

lemire commented 5 years ago

Upstream, simdjson will provide runtime dispatch. So this should not be necessary in the future.

cc @ioioioio

luizperes commented 5 years ago

I have updated to 0.2.0. Could you confirm that @lemire and @ioioioio?

lemire commented 5 years ago

@luizperes I think that your code still check for AVX2. You don't need that anymore.

What you do (check for AVX2 and then bail out) should work fine if you are not on ARM, but you could weaken quite a bit the checks.

luizperes commented 5 years ago

So, would it suffice if we said we don't support 32-bit at all?

I will likely use the same directives simdjon is using then, good to know, thanks!

lemire commented 5 years ago

Right... and to be prudent check for sse 4.2 and clmul on x64.

luizperes commented 4 years ago

I just realized I did not update that (with #23). Hi @lemire, is it correct that we don't need to use the flags -mavx2 anymore?

lemire commented 4 years ago

Please be advised that we no longer require SSE 4.2. The upcoming version of simdjson will run everywhere.

lemire commented 4 years ago

@luizperes You should update to version 0.3 in a few days. You will just compile without any flag. It will run everywhere.

luizperes commented 4 years ago

PR #27 fixes this issue