search

reactphp / filesystem

Evented filesystem access.
MIT License
135 stars 40 forks source link

Not opening/reading all files #70

Open petsoukos opened 4 years ago

petsoukos commented 4 years ago

Description

Trying to read 10 files from disk usually ends up skipping some of them randomly with error

Unknown error calling: "eio_open"

Takes for ever to load with the ChildProcess adapter. strlen() also shows different bytes each time (might be different issue)

OS/PHP versions

Sample script

https://gist.github.com/petsoukos/2cc5f7cc3181efbaab12d43fbe325644

Create some random files with sizes over 3MB up to 150MB and store em in a "downloads" folder or adjust the script to point to your files.

Sample output

$ php Processor.php 
Bytes loaded for file file10.xml: 27164772
Bytes loaded for file file9.xml: 27159323
Bytes loaded for file file8.xml: 175390820
Bytes loaded for file file7.xml: 27099273
Bytes loaded for file file6.xml: 27145682
Bytes loaded for file file5.xml: 27080067
Bytes loaded for file file4.xml: 306454628
Opening file3.xml produced some error: Unknown error calling "eio_open"
Bytes loaded for file file2.xml: 227803236
Bytes loaded for file file1.xml: 27085574
$

$ php Processor.php 
Bytes loaded for file file10.xml: 30176743
Bytes loaded for file file9.xml: 30171294
Bytes loaded for file file8.xml: 178394599
Bytes loaded for file file7.xml: 30111244
Bytes loaded for file file6.xml: 30157653
Bytes loaded for file file5.xml: 30097545
Bytes loaded for file file4.xml: 309466599
Bytes loaded for file file3.xml: 30083846
Bytes loaded for file file2.xml: 230831591
Bytes loaded for file file1.xml: 30070147
$

$ php Processor.php 
Bytes loaded for file file10.xml: 27164772
Bytes loaded for file file9.xml: 27159323
Bytes loaded for file file8.xml: 175374436
Opening file7.xml produced some error: Unknown error calling "eio_open"
Bytes loaded for file file6.xml: 27145682
Bytes loaded for file file5.xml: 27091081
Bytes loaded for file file4.xml: 306454628
Bytes loaded for file file3.xml: 27077382
Bytes loaded for file file2.xml: 227795044
Bytes loaded for file file1.xml: 27063683
$
ghost commented 4 years ago

The eio error is known #43...

That child process takes long to read large files is unfortunate but it's how it is. You need to tell a child process what to do, the child process reads the file, squeezes the data through a socket and the main process needs to read the socket.

Using a buffer and strlen is inefficient and can lead to wrong results, use stat instead. Even eio shows here a wrong result, it reads 71MB but the file is 98MB (ls -l). And child process fetches over 200MB....

petsoukos commented 4 years ago

Yes, this is unfortunate.

I put this aside for now and created what I required with another solution (which isn't a reactor pattern)

But, I would still like to give this a shot and do it with ReactPHP.

ghost commented 4 years ago

The changing numbers of filesize is related to how variables work. I've changed your one variable $buffer to an array of integers (representing the bytes read) - so for each file there's an element in the array. This change made the filesize correct for each file. Previously due to the sharing variable (PHP is not lexical scoped) the filesize was non-deterministic and incorrect.