search

sibnerian / electron-promise-ipc

Promise-flavored IPC calls in Electron. 100% test coverage.
MIT License
68 stars 10 forks source link

Any known limit for data transfert through IPC ? #22

Closed lgodard closed 4 years ago

lgodard commented 4 years ago

Hi

First thanks for electron-promise-ipc

I need to send a huge array to IPC main in order to dump it on disk (huge means 167 Mb stringyfied)

It ends with electron devtools error (disconnected) or a blank screen

Is there a known limit size when sending data via electron-promise-ipc ? Any suggestion to pass such big arguments through IPC ?

Thanks in advance

sibnerian commented 4 years ago

It ends with electron devtools error (disconnected) or a blank screen

My guess is that you're running up against the limits of Electron's IPC mechanism -- electron-promise-ipc is a thin wrapper over that.

Is there a known limit size when sending data via electron-promise-ipc?

I've never experimented with the size limit, but it looks like you found it!

Any suggestion to pass such big arguments through IPC ?

Your best bet is to split your data into several chunks. Here's how I'd do it (there are multiple schemes you could use of course):

  1. Stringify your payload and break it into a series of equal-sized chunks. You'll have to experiment with the chunk size that makes sense (i.e. doesn't crash the app). 1MB is probably safe.
  2. Start with a message describing your payload (i.e. how many characters total, and how many parts you're planning to send). Include a transfer ID.
  3. When that initial message is acknowledged (the promise resolves), kick off a series of sequential IPC calls, one for each chunk. a. Make sure that you mark each chunk with its transfer ID sequence number! b. Sequential send is optional, but probably makes sense for your use case, since it means you can put each chunk on disk immediately and avoid keeping it all in memory.
  4. On the other side of the transfer, you'll receive the initial message and prepare to receive N chunks for a certain transfer ID. a. Since you're sending them sequentially, you can append each one to your dump file to avoid keeping them all in memory.
  5. Once all of your chunk IPCs are received, the dump file is completed!
lgodard commented 4 years ago

Hi thanks a lot for your response and suggestion i think is a correct way to experiment thanks again for your electron-promise-ipc module Laurent