Resolve array size limitations (mostly OK now on Node 10+)

publiclab / image-sequencer-app

An Image Processing server based on image-sequencer

GNU General Public License v3.0

7 stars 5 forks source link

Resolve array size limitations (mostly OK now on Node 10+) #12

Open jywarren opened 5 years ago

jywarren commented 5 years ago

@tech4GT noted:

Hmm, looks like we are beyond the allowed array length while making a canvas this big! We would have to look for a workaround for this. @jywarren Any ideas?

RangeError: Invalid typed array length: 13650935808

@tech4GT opened this asking how to handle larger arrays: https://github.com/scijs/get-pixels/issues/46

Noting that https://github.com/scijs/get-pixels and https://github.com/scijs/save-pixels use https://github.com/scijs/ndarray for arrays.

And the above error references a Uint8Array - so a typed array. https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Errors/Invalid_array_length

There seem to be lots of ndarray implementations:

https://github.com/scijs/ndarray/wiki/ndarray-module-list#core-module

Could one of those help us?

Also noting this lib designed to work around memory limits on untyped arrays: https://www.npmjs.com/package/lds

tech4GT commented 5 years ago

@jywarren Just for the sake of completeness, is there some sort of upper limit on the size of the images we may get here?

jywarren commented 5 years ago

haha... uhhhh i mean i've seen people try to export 50,000x50,000px images!

tech4GT commented 5 years ago

Woah! Yeah we'll have to think this one out for sure! Image sequencer can't generate a canvas this big in the first place, and we might have multiple of these! 🤯

jywarren commented 5 years ago

haha! I am not sure one of these has successfully exported, to be honest, even in Ruby! But, one thing to think of is potentially rendering in segments - like, breaking it into max 10,000px^2 squares and rendering each square independently. It's complicated. But we can think about this kind of optimization much later.

To refocus back on the issue at hand here, what dimensions are we dealing with in more conventional maps? I would guess we want to be able to export in the range of 5000-10,000px square, max.

tech4GT commented 5 years ago

Alright! As long as we have only a few of these images and they have a sizeable overlap, I think we can handle it with mininal scaling! I was able to handle the ceres -- 2 map without any scaling of the input!

jywarren commented 5 years ago

OK, but that's one of the smaller maps out there... see https://github.com/publiclab/mapknitter-exporter-sinatra/issues/23 for some of the ones we're outputting in Ruby!

tech4GT commented 5 years ago

Great! Thanks I'll try those out tomorrow, along with the documentation and tests! It's pretty late here haha! I was wondering if we can discuss the multiprocessor scaling as well right now so that I can code it up tomorrow! 😅

jywarren commented 5 years ago

Put it this way - what is the max size arrays can be? MDN says:

The length property of an Array or an ArrayBuffer is represented with an unsigned 32-bit integer, that can only store values which are in the range from 0 to 2^32-1.

So that's: 4294967296 or 4,294,967,296 -- four billion or so. Not sure but imagine RGBA is 4 values, so a billion pixels, and the square root of that is 32,768 -

So is 32768x32768 roughly our canvas max?

tech4GT commented 5 years ago

Actually it's less for typed arrays, I wasn't able to find the exact value but 2^26 works and 2^27 doesn't. Maybe that is a machine dependent thing but that's what happened on my mac!

tech4GT commented 5 years ago

Aha! @jywarren this is dependent on the version of node!! I tried this with node 10 and it allows up to 2^30 !!!

tech4GT commented 5 years ago

I was using node 9 earlier!

jywarren commented 5 years ago

oh wow! What had it been earlier? Anyways, that's great.

On Tue, May 28, 2019 at 1:35 PM Varun Gupta notifications@github.com wrote:

I was using node 9 earlier!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/image-sequencer-app/issues/12?email_source=notifications&email_token=AAAF6JZWN4ICSEPOEHVQK3DPXVUMDA5CNFSM4HQFDYY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWM334Q#issuecomment-496614898, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAF6J674QTGB3OFOHKTUMLPXVUMDANCNFSM4HQFDYYQ .

tech4GT commented 5 years ago

Yeah I think 2^30 is a big enough canvas for our purpose!! Great!! :tada:

jywarren commented 5 years ago

OK I'll mark this as temporarily resolved...