terminus wp cli media import not working if using wildcard for filenames

carl-alberto commented 5 years ago

Expected behavior

As stated in the wp media import examples here https://developer.wordpress.org/cli/commands/media/import/, you can import bulk file using the wildcard examples but it seems it is not working, single file media import works fine.

Actual behavior

This works for a single file on an absolute filepath in server: terminus wp testenvname.dev -- media import '/srv/bindings/daa068ccf4fXXXXXXX/code/wp-content/uploads/test/test1.png' --debug

Tried uploading from local to server: terminus wp testenvname.dev -- media import /Users/carlalberto/Desktop/*.png --debug

Debug (bootstrap): Required file from config: /opt/pantheon/wp-launch-check-0.x/wp-launch-check.phar (0.016s)
Debug (bootstrap): Using default global config: /srv/bindings/daa068ccf4fXXXXXXX/.wp-cli/config.yml (0.054s)
Debug (bootstrap): No project config found (0.054s)
Debug (bootstrap): argv: /opt/pantheon/wp-cli1/wp-cli.phar --require=/opt/pantheon/wp-launch-check-0.x/wp-launch-check.phar media import /Users/carlalberto/Desktop/test1.png /Users/carlalberto/Desktop/test2.png --debug (0.054s)
Debug (bootstrap): ABSPATH defined: /srv/bindings/daa068ccf4fXXXXXXX/code/ (0.054s)
Debug (bootstrap): Set URL: dev-carltest1.pantheonsite.io (0.054s)
Debug (bootstrap): Begin WordPress load (0.054s)
Debug (bootstrap): wp-config.php path: /srv/bindings/daa068ccf4fXXXXXXX/code/wp-config.php (0.054s)
Debug (commandfactory): No doc comment for redis. (0.315s)
Debug (bootstrap): Loaded WordPress (0.381s)
Debug (bootstrap): Running command: media import (0.381s)
Warning: Unable to import file '/Users/carlalberto/Desktop/test1.png'. Reason: File doesn't exist.
Warning: Unable to import file '/Users/carlalberto/Desktop/test2.png'. Reason: File doesn't exist.
Error: No items imported.

These do not work, using no quote, with quote and double quotes for the path and using *.png, *.* and **\/*.png terminus wp carltest1.dev -- media import "/srv/bindings/daa068ccf4fXXXXXXX/code/wp-content/uploads/test2/*.png" --debug

terminus wp carltest1.dev -- media import "/srv/bindings/daa068ccf4fXXXXXXX/code/wp-content/uploads/test2/*.*" --debug

terminus wp carltest1.dev -- media import "/srv/bindings/daa068ccf4fXXXXXXX/code/wp-content/uploads/test2/**\/*.png" --debug

Debug (bootstrap): Required file from config: /opt/pantheon/wp-launch-check-0.x/wp-launch-check.phar (0.015s)
Debug (bootstrap): Using default global config: /srv/bindings/daa068ccf4fXXXXXXX/.wp-cli/config.yml (0.071s)
Debug (bootstrap): No project config found (0.071s)
Debug (bootstrap): argv: /opt/pantheon/wp-cli1/wp-cli.phar --require=/opt/pantheon/wp-launch-check-0.x/wp-launch-check.phar media import /srv/bindings/daa068ccf4fXXXXXXX/code/wp-content/uploads/test2/*.* --debug -vvv (0.071s)
Debug (bootstrap): ABSPATH defined: /srv/bindings/daa068ccf4fXXXXXXX/code/ (0.071s)
Debug (bootstrap): Set URL: dev-carltest1.pantheonsite.io (0.071s)
Debug (bootstrap): Begin WordPress load (0.071s)
Debug (bootstrap): wp-config.php path: /srv/bindings/daa068ccf4fXXXXXXX/code/wp-config.php (0.071s)
Debug (commandfactory): No doc comment for redis. (0.306s)
Debug (bootstrap): Loaded WordPress (1.16s)
Debug (bootstrap): Running command: media import (1.161s)
Warning: Unable to import file '/srv/bindings/daa068ccf4fXXXXXXX/code/wp-content/uploads/test2/*.*'. Reason: File doesn't exist.
Warning: Unable to import file '-vvv'. Reason: File doesn't exist.
Error: No items imported.
 [notice] Command: carltest1.dev -- wp media import /srv/bindings/daa068ccf4fXXXXXXX/code/wp-content/uploads/test2/*.* [Exit: 1]
 [error]

https://screencast.com/t/29hv3ZbQr

terminus self:info                                                                
 ------------------------- ------------------------------------------------------------- 
  PHP binary                /usr/local/Cellar/php@7.1/7.1.23/bin/php                     
  PHP version               7.1.23                                                       
  php.ini used              /usr/local/etc/php/7.1/php.ini                               
  Terminus project config                                                                
  Terminus root dir         /Users/carlalberto/vendor/vendor/pantheon-systems/terminus   
  Terminus version          1.9.0                                                        
  Operating system          Darwin Kernel Version 16.7.0: Thu Dec 20 21:53:35 PST 2018;  
                            root:xnu-3789.73.31~1/RELEASE_X86_64                         
 ------------------------- -------------------------------------------------------------

Another note is it working fine using the internal btool wp media import using *.jpg but not **\/*.png https://screencast.com/t/uGb7a2AZ2

charmoney commented 3 years ago

Encountered this exact issue on Terminus 2.5.0 today. The WP CLI media import works perfectly via terminus for full paths but fails with glob wildcards, both for a single directory extension glob /path/to/files/pdfs/*.pdf and a multiple directory glob /path/to/**/*.pdf.

No combination of quotes or backslash escapes makes a difference.

Identical behavior from a mac and a linux server.

terminus self:info
 ------------------------- ---------------------------------------------------------------------------------------------------
  PHP binary                /usr/local/Cellar/php@7.4/7.4.14_1/bin/php
  PHP version               7.4.14
  php.ini used              /usr/local/etc/php/7.4/php.ini
  Terminus project config
  Terminus root dir         /Users/harmoney/terminus/vendor/pantheon-systems/terminus
  Terminus version          2.5.0
  Operating system          Darwin Kernel Version 19.6.0: Tue Jan 12 22:13:05 PST 2021; root:xnu-6153.141.16~1/RELEASE_X86_64
 ------------------------- ---------------------------------------------------------------------------------------------------

terminus self:info
 ------------------------- ------------------------------------------------
  PHP binary                /usr/bin/php7.3
  PHP version               7.3.26-1+ubuntu18.04.1+deb.sury.org+1
  php.ini used              /etc/php/7.3/cli/php.ini
  Terminus project config
  Terminus root dir         /opt/composer/vendor/pantheon-systems/terminus
  Terminus version          2.5.0
  Operating system          #139-Ubuntu SMP Mon Jan 18 17:38:24 UTC 2021
 ------------------------- ------------------------------------------------

reidelliott commented 3 years ago

Did either of you find a solution for this?

charmoney commented 3 years ago

@reidelliott it's incredibly wasteful and tedious but yes. The wp media import command accepts multiple paths, which can be the full explicit path to the file that terminus handles just fine.

Use the *nix find command to get the relative paths to all files in the directory I wanted to wildcard import from find wp-content/uploads/archive/ -type f
Copy that to a text editor and surround each filename in single quotes to handle spaces. An editor with syntax highlighting helps identify filenames that themselves contain single quotes. Deal with those by either switching to double quotes or escaping the filename quote.
Concatenate many quoted paths onto a single line, space separated, then toss the terminus command at the front. terminus wp $SITE.$ENV -- media import --skip-copy --preserve-filetime 'wp-content/uploads/archive/...pdf' 'wp-content/uploads/archive/...pdf'

I was able to import the site's 1800ish files in a few batches of up to 500 each much faster than importing them individually, since all those files only required terminus overhead once. By the time I gradually scaled up to 500 file paths in a single command, I was close enough to done I just stayed at that. I suspect the command could handle even more paths without a problem.

Warning: Terminus seems to mangle some UTF8 characters used in filenames. Around 10 showed up in the site's 1800ish files. I just sent those URLs to the site content folks and recommended they download the files, remove fancy quotes & other non-standard characters, and re-upload manually. There weren't enough to be worth dev troubleshooting time.

mitchellkrogza commented 3 years ago

Ridiculous, their docs say wp media import ~/Pictures/**\/*.jpg but it doesn't work at all

zyend commented 12 months ago

same issue - end up using a loop when a wildcard could of solve it.

pantheon-systems / terminus

terminus wp cli media import not working if using wildcard for filenames #1938

Expected behavior

Actual behavior