Open eduardoarandah opened 4 years ago
Same issue here. Thanks for the proposed solution.
I cannot replicate the above, the command just works fine for me for both v2.4.0 and for latest dev.
Can you post the output of the following commands for me to get more information about your environment?
ls -ls $(which wp)
ls -ls $(which curl)
I suspect there's a script in-between that messes up the IO.
I cannot replicate the above, the command just works fine for me for both v2.4.0 and for latest dev. Can you post the output of the following commands for me to get more information about your environment?
Howdy,
www-data@e82a29c5348e:~$ ls -ls $(which wp)
5440 -rwxr-xr-x 1 root root 5568133 Feb 18 08:10 /usr/local/bin/wp
www-data@e82a29c5348e:~$ ls -ls $(which curl)
236 -rwxr-xr-x 1 root root 239848 Nov 30 15:49 /usr/bin/curl
www-data@e82a29c5348e:~$ curl --version
curl 7.68.0 (x86_64-pc-linux-gnu) libcurl/7.68.0 OpenSSL/1.1.1f zlib/1.2.11 brotli/1.0.9 libidn2/2.3.0 libpsl/0.21.0 (+libidn2/2.3.0) libssh/0.9.3/openssl/zlib nghttp2/1.41.0 librtmp/2.3
Release-Date: 2020-01-08
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS brotli GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets
www-data@e82a29c5348e:~$ wp --version
WP-CLI 2.4.0
This is running inside of Docker
www-data@e82a29c5348e:~/multi$ curl -Ns http://loripsum.net/api/5 > foo.txt
www-data@e82a29c5348e:~/multi$ cat foo.txt
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Estne, quaeso, inquam, sitienti in bibendo voluptas? Sapiens autem semper beatus est et est aliquando in dolore; Hic nihil fuit, quod quaereremus. Habes, inquam, Cato, formam eorum, de quibus loquor, philosophorum. </p>
(...)
www-data@e82a29c5348e:~/multi$ curl -Ns http://loripsum.net/api/5 | grep Lorem
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Ut alios omittam, hunc appello, quem ille unum secutus est. Conferam tecum, quam cuique verso rem subicias; Sed haec quidem liberius ab eo dicuntur et saepius. Illum mallem levares, quo optimum atque humanissimum virum, Cn. Sit hoc ultimum bonorum, quod nunc a me defenditur; Duo Reges: constructio interrete. Paria sunt igitur. Sed haec quidem liberius ab eo dicuntur et saepius. Itaque in rebus minime obscuris non multus est apud eos disserendi labor. Sed quanta sit alias, nunc tantum possitne esse tanta. Sin tantum modo ad indicia veteris memoriae cognoscenda, curiosorum. </p>
www-data@e82a29c5348e:~/multi$ curl -Ns http://loripsum.net/api/5 | wp post generate --post_content --count=10
Error: The parameter `post_content` reads from STDIN.
I'm available for a quick anydesk call if that helps
STDIN seems to be populated in PHP:
www-data@e82a29c5348e:~/single_tests$ echo "foo" | /usr/bin/php -r 'echo file_get_contents("php://stdin");'
foo
www-data@e82a29c5348e:~/single_tests$ echo "foo" | /usr/bin/php -r 'echo stream_get_contents(STDIN);'
foo
This works:
www-data@e82a29c5348e:~/single_tests$ echo "Foo" | wp post generate --post_content --count=10
Generating posts 100% [=====================================================================================] 0:00 / 0:00
So it means it's something with my CURL? Even though this works, too.
www-data@e82a29c5348e:~/single_tests$ curl -Ns http://loripsum.net/api/5 | cat /dev/stdin
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Negat enim summo bono afferre incrementum diem. Hoc etsi multimodis reprehendi potest, tamen accipio, quod dant. ALIO MODO. Aliter homines, aliter philosophos loqui putas oportere? Cum ageremus, inquit, vitae beatum et eundem supremum diem, scribebamus haec. Atqui haec patefactio quasi rerum opertarum, cum quid quidque sit aperitur, definitio est. Duo Reges: constructio interrete. Ut enim consuetudo loquitur, id solum dicitur honestum, quod est populari fama gloriosum. Utrum igitur tibi litteram videor an totas paginas commovere? Scio enim esse quosdam, qui quavis lingua philosophari possint; </p>
And, this works:
curl -Ns http://loripsum.net/api/5 | php -r 'echo file_get_contents("php://stdin");'
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Haec et tu ita posuisti, et verba vestra sunt. Cur deinde Metrodori liberos commendas? </p>
(...)
Something in this code is bailing the execution before it can try to read from STDIN, only when feeded by CURL: https://github.com/wp-cli/entity-command/blob/209d1c585d8a27c0b436b1b80c58326f789a8442/src/WP_CLI/Entity/Utils.php#L12
www-data@e82a29c5348e:~/single_tests$ curl -Ns http://loripsum.net/api/5 | php -r 'echo $handle = fopen( "php://stdin", "r" );$read = array( $handle );$write = null;$except = null;$streams = stream_select( $read, $write, $except, 0 );fclose( $handle ); var_dump($streams);'
Resource id #5Command line code:1:
int(0)
www-data@e82a29c5348e:~/single_tests$ echo "Foo" | php -r 'echo $handle = fopen( "php://stdin", "r" );$read = array( $handle );$write = null;$except = null;$streams = stream_select( $read, $write, $except, 0 );fclose( $handle ); var_dump($streams);'
Resource id #5Command line code:1:
int(1)
Aha! The problem is the timeout: stream_select( $read, $write, $except, 0 );
=> stream_select( $read, $write, $except, 10 );
www-data@e82a29c5348e:~/single_tests$ curl -Ns http://loripsum.net/api/5 | php -r 'echo $handle = fopen( "php://stdin", "r" );$read = array( $handle );$write = null;$except = null;$streams = stream_select( $read, $write, $except, 10 );fclose( $handle ); var_dump($streams);'
Resource id #5Command line code:1:
int(1)
I've added a timeout of 10s, the stream returns as expected with Curl now.
10s is just the timeout, but it seems it returns as soon as ready, if it will be ready at all:
www-data@e82a29c5348e:~/single_tests$ time curl -Ns http://loripsum.net/api/5 | php -r 'echo $handle = fopen( "php://stdin", "r" );$read = array( $handle );$write = null;$except = null;$streams = stream_select( $read, $write, $except, 10 );fclose( $handle ); var_dump($streams);'
Resource id #5Command line code:1:
int(1)
real 0m0.450s
user 0m0.008s
sys 0m0.013s
@schlessera Can you please take a look at the debugging process and proposed solution above? :point_up:
Essentially, increasing the timeout here:
One downside of the timeout approach is that if there's no STDIN, it will take 10s for the command to fail:
www-data@e82a29c5348e:~/single_tests$ time php -r 'echo $handle = fopen( "php://stdin", "r" );$read = array( $handle );$write = null;$except = null;$streams = stream_select( $read, $write, $except, 10 );fclose( $handle ); var_dump($streams);'
Resource id #5Command line code:1:
int(0)
real 0m10.057s
user 0m0.027s
sys 0m0.021s
I'm not sure if there are better ways to tackle this. Any timeout below 1s is returning an empty stream for me, it seems the timeout is directly related to the time it takes for curl to feed the data, so it's a combination of issues between curl and the no timeout.
So that's why pre-computing the result of curl works, in the example of the original thread:
content=$(curl -Ns http://loripsum.net/api/5)
www-data@e82a29c5348e:~/single_tests$ time echo $content | php -r 'echo $handle = fopen( "php://stdin", "r" );$read = array( $handle );$write = null;$except = null;$streams = stream_select( $read, $write, $except, 0 );fclose( $handle ); var_dump($streams);'
Resource id #5Command line code:1:
int(1)
real 0m0.018s
user 0m0.007s
sys 0m0.013s
What I find most curious is that the STDIN is there when PHP starts execution, so the timeout should be unnecessary:
www-data@e82a29c5348e:~/single_tests$ time curl -Ns http://loripsum.net/api/50 | php -r 'echo substr(file_get_contents("php://stdin"), 0, 100);'
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Quamquam haec quidem praeposita recte et
real 0m1.056s
user 0m0.004s
sys 0m0.033s
Maybe just read file_get_contents("php://stdin")
and if it's empty bail? Instead of using stream_select
I'm getting the same error.
System:
OS: Linux 5.4.0-126-generic #142-Ubuntu SMP Fri Aug 26 12:12:57 UTC 2022 x86_64
Shell: /bin/bash
PHP binary: /usr/bin/php8.1
PHP version: 8.1.10
php.ini used: /etc/php/8.1/cli/php.ini
MySQL binary: /usr/bin/mysql
MySQL version: mysql Ver 8.0.30-0ubuntu0.20.04.2 for Linux on x86_64 ((Ubuntu))
SQL modes:
WP-CLI root dir: phar://wp-cli.phar/vendor/wp-cli/wp-cli
WP-CLI vendor dir: phar://wp-cli.phar/vendor
WP_CLI phar path: /home/milana
WP-CLI packages dir: /home/milana/.wp-cli/packages/
WP-CLI global config: /home/milana/.wp-cli/config.yml
WP-CLI project config:
WP-CLI version: 2.6.0
Command:
curl -N https://baconipsum.com/api/?type=meat-and-filler | wp post generate --post_content --count=36
Error:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Error: The parameter `post_content` reads from STDIN.
100 1536 100 1536 0 0 1545 0 --:--:-- --:--:-- --:--:-- 1543
curl: (23) Failed writing body (0 != 1536)
But this one wors:
curl -N http://loripsum.net/api/5 | wp post generate --post_content --count=10
Could be the API issue
@zzap Please try this:
curl -N https://baconipsum.com/api/?type=meat-and-filler > foo.txt && cat foo.txt | wp post generate --post_content && rm foo.txt
With the approach above, we are essentially pre-caching the remote response in a local file and then feeding it to WP CLI, essentially minimizing delays that could be causing this issue.
I don't know exactly what's happening, but for a definitive solution I'd play around with increasing the stream_select
timeout in https://github.com/wp-cli/entity-command/blob/209d1c585d8a27c0b436b1b80c58326f789a8442/src/WP_CLI/Entity/Utils.php#L17 from 0
to 1
or more
I encountered this issue, as well. The workaround @Luc45 suggested worked.
Would a permanent solution be to add an optional --timeout
parameter? Looking at the stream_select()
docs:
The seconds and microseconds together form the timeout parameter, seconds specifies the number of seconds while microseconds the number of microseconds. The timeout is an upper bound on the amount of time that stream_select() will wait before it returns. If seconds and microseconds are both set to 0, stream_select() will not wait for data - instead it will return immediately, indicating the current status of the streams.
So the 0
in the seconds parameter means it will not work for anything except a file redirect or other instantaneous piping, which is good for the majority of use cases, but breaks things like the example in the command help: curl -N https://loripsum.net/api/5 | wp post generate --post_content --count=10
The --timeout
option can default to 0
, creating the default behavior, but can be updated for cases where there will be a delay from STDIN
via curl
or other lengthy command.
Bug Report
Describe the current, buggy behavior
curl -Ns http://loripsum.net/api/5 | wp post generate --post_content --count=10
Gives me:
Error: The parameter
post_contentreads from STDIN.
Curl works fine:
Describe how other contributors can replicate this bug
Describe what you would expect as the correct outcome
Generate posts
Let us know what environment you are running this on
Provide a possible solution
If you happen to have a suggestion on how to fix this bug, please tell us in here.
Just leave this section out if you don't know how to fix it.
Provide additional context/Screenshots
this works:
echo "hello world" | wp post generate --post_content --count=10
This works: