PHAR fails when relative cache || build path is used

[ptomulik]

With 5.2.1, the phar version fails to create directories within cache or build path, when relative cache/build paths are used in config. Here is an example run:

ptomulik@barakus:$ php bin/doctum.phar update --force -vvv docs/doctum-relative.conf.php 
 Updating project 

Version main
-------------

In Filesystem.php line 105:

  [Symfony\Component\Filesystem\Exception\IOException]      
  Failed to create "docs/cache/html": mkdir(): File exists  

Exception trace:
  at phar:///tmp/test/bin/doctum.phar/vendor/symfony/filesystem/Filesystem.php:105
 Symfony\Component\Filesystem\Filesystem->mkdir() at phar:///tmp/test/bin/doctum.phar/src/Project.php:380
 Doctum\Project->flushDir() at phar:///tmp/test/bin/doctum.phar/src/Project.php:431
 Doctum\Project->prepareDir() at phar:///tmp/test/bin/doctum.phar/src/Project.php:374
 Doctum\Project->getCacheDir() at phar:///tmp/test/bin/doctum.phar/src/Store/JsonStore.php:76
 Doctum\Store\JsonStore->getStoreDir() at phar:///tmp/test/bin/doctum.phar/src/Store/JsonStore.php:64
 Doctum\Store\JsonStore->flushProject() at phar:///tmp/test/bin/doctum.phar/src/Project.php:464
 Doctum\Project->parseVersion() at phar:///tmp/test/bin/doctum.phar/src/Project.php:125
 Doctum\Project->update() at phar:///tmp/test/bin/doctum.phar/src/Console/Command/Command.php:183
 Doctum\Console\Command\Command->update() at phar:///tmp/test/bin/doctum.phar/src/Console/Command/UpdateCommand.php:54
 Doctum\Console\Command\UpdateCommand->execute() at phar:///tmp/test/bin/doctum.phar/vendor/symfony/console/Command/Command.php:255
 Symfony\Component\Console\Command\Command->run() at phar:///tmp/test/bin/doctum.phar/vendor/symfony/console/Application.php:1009
 Symfony\Component\Console\Application->doRunCommand() at phar:///tmp/test/bin/doctum.phar/vendor/symfony/console/Application.php:273
 Symfony\Component\Console\Application->doRun() at phar:///tmp/test/bin/doctum.phar/vendor/symfony/console/Application.php:149
 Symfony\Component\Console\Application->run() at phar:///tmp/test/bin/doctum.phar/bin/doctum-binary.php:26
 include() at /tmp/test/bin/doctum.phar:9

update [--only-version ONLY-VERSION] [--force] [--output-format OUTPUT-FORMAT] [--no-progress] [--ignore-parse-errors] [--] <config>

The problem does not appear with non-phar version. Absolute paths work well.

I attach an archive containing a minimal example. The issue looks very strange and may be related to Symfony\Filesystem.

test.tar.gz

[ptomulik]

The config within the minimal example is the following

<?php

use Doctum\Doctum;
use Symfony\Component\Finder\Finder;

$iterator = Finder::create()
  ->files()
  ->name("*.php")
  ->in(['src']);

return new Doctum($iterator, [
  'theme'     => 'default',
  'title'     => 'API Documentation',
  'build_dir' => 'docs/build/html',
  'cache_dir' => 'docs/cache/html',
]);

[williamdes]

Hi!

Did this work before? With 5.0.0 Phar?

[ptomulik]

Hi!

Did this work before? With 5.0.0 Phar?

No.

ptomulik@barakus:$ php bin/doctum-5.0.0.phar update --force -vvv docs/doctum-relative.conf.php 
#!/usr/bin/env php
 Updating project 

Version main

In Filesystem.php line 100:

  [Symfony\Component\Filesystem\Exception\IOException]      
  Failed to create "docs/cache/html": mkdir(): File exists  

Exception trace:
  at phar:///tmp/test/bin/doctum-5.0.0.phar/vendor/symfony/filesystem/Filesystem.php:100
 Symfony\Component\Filesystem\Filesystem->mkdir() at phar:///tmp/test/bin/doctum-5.0.0.phar/src/Project.php:343
 Doctum\Project->flushDir() at phar:///tmp/test/bin/doctum-5.0.0.phar/src/Project.php:394
 Doctum\Project->prepareDir() at phar:///tmp/test/bin/doctum-5.0.0.phar/src/Project.php:337
 Doctum\Project->getCacheDir() at phar:///tmp/test/bin/doctum-5.0.0.phar/src/Store/JsonStore.php:76
 Doctum\Store\JsonStore->getStoreDir() at phar:///tmp/test/bin/doctum-5.0.0.phar/src/Store/JsonStore.php:64
 Doctum\Store\JsonStore->flushProject() at phar:///tmp/test/bin/doctum-5.0.0.phar/src/Project.php:427
 Doctum\Project->parseVersion() at phar:///tmp/test/bin/doctum-5.0.0.phar/src/Project.php:108
 Doctum\Project->update() at phar:///tmp/test/bin/doctum-5.0.0.phar/src/Console/Command/Command.php:79
 Doctum\Console\Command\Command->update() at phar:///tmp/test/bin/doctum-5.0.0.phar/src/Console/Command/UpdateCommand.php:53
 Doctum\Console\Command\UpdateCommand->execute() at phar:///tmp/test/bin/doctum-5.0.0.phar/vendor/symfony/console/Command/Command.php:258
 Symfony\Component\Console\Command\Command->run() at phar:///tmp/test/bin/doctum-5.0.0.phar/vendor/symfony/console/Application.php:911
 Symfony\Component\Console\Application->doRunCommand() at phar:///tmp/test/bin/doctum-5.0.0.phar/vendor/symfony/console/Application.php:264
 Symfony\Component\Console\Application->doRun() at phar:///tmp/test/bin/doctum-5.0.0.phar/vendor/symfony/console/Application.php:140
 Symfony\Component\Console\Application->run() at phar:///tmp/test/bin/doctum-5.0.0.phar/bin/doctum.php:15
 include() at /tmp/test/bin/doctum-5.0.0.phar:9

update [--only-version ONLY-VERSION] [--force] [--] <config>

[williamdes]

I am very happy this also fails (I did not break anything) I will investigate, thank you so much for the example that I will surely add to my unit tests

[ptomulik]

The issue looks quite strange to me. The exception appears to be thrown from within symfony, from mkdir() function (not sure, this is exactly same version of symfony/filesystem you use to create phar). The check in line 92 should prevent the "File exists" exception, except there exists a file with the same name as $dir (and it's not a directory).

[ptomulik]

Some additional details:

ptomulik@barakus:$ php --version
PHP 7.4.11 (cli) (built: Oct  6 2020 10:34:39) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Zend OPcache v7.4.11, Copyright (c), by Zend Technologies
    with Xdebug v2.9.6, Copyright (c) 2002-2020, by Derick Rethans

The same happens with php 7.3.

[ptomulik]

I investigated it further, with the following script (call it bin/test.php)

#!/usr/bin/env php
<?php

if (count($argv) > 1) {
    printf("file_exists(%s): %s\n", $argv[1], file_exists($argv[1]) ? 'true' : 'false');
    printf("is_dir(%s): %s\n", $argv[1], is_dir($argv[1]) ? 'true' : 'false');
}

First ran it normally providing relative name of an existing directory:

ptomulik@barakus:$ bin/test.php foo/
is_file(foo/): false
is_dir(foo/): true

then made a phar out of bin/test.php using your phar-generator-script.sh (with slight modification to use bin/test.php instead of bin/doctum.php). Running the phar version results with

ptomulik@barakus:$ php build/test.phar foo/
#!/usr/bin/env php
file_exists(foo/): false
is_dir(foo/): false

It looks, like file_exists() & is_dir() do not work well with relative paths from phar.

[williamdes]

Thank you, this is precious details Maybe some kind of resolver should be used in phars If you have some documentation that would be interesting

[ptomulik]

I know very little about phars. The particular issue seems to have the following cause - within our PHAR, some functions seem to prepend phar:///... to relative paths, even if the getcwd() returns a path from real filesystem. The directories included in PHAR are detected with relative names, the directories not included are not, even if they're present in real filesystem at runtime.

#!/usr/bin/env php
<?php

printf("getcwd(): %s\n", getcwd());
if (count($argv) > 1) {
    printf("realpath(%s): %s\n", $argv[1], realpath($argv[1]));
    printf("file_exists(%s): %s\n", $argv[1], file_exists($argv[1]) ? 'true' : 'false');
    printf("is_dir(%s): %s\n", $argv[1], is_dir($argv[1]) ? 'true' : 'false');
}

ptomulik@barakus:$ php build/test.phar bin/
#!/usr/bin/env php
getcwd(): /tmp/pht
realpath(bin/): /tmp/pht/bin
file_exists(bin/): true
is_dir(bin/): true
ptomulik@barakus:$ php build/test.phar foo/
#!/usr/bin/env php
getcwd(): /tmp/pht
realpath(foo/): /tmp/pht/foo
file_exists(foo/): false
is_dir(foo/): false
ptomulik@barakus:$ ls -lah
razem 396K
drwxr-xr-x  6 ptomulik ptomulik 4,0K 12-09 09:10 .
drwxrwxrwt 28 root     root     372K 12-09 09:25 ..
drwxr-xr-x  2 ptomulik ptomulik 4,0K 12-09 09:23 bin
drwxr-xr-x  2 ptomulik ptomulik 4,0K 12-09 09:25 build
drwxr-xr-x  2 ptomulik ptomulik 4,0K 12-09 09:28 foo
drwxr-xr-x  2 ptomulik ptomulik 4,0K 12-09 09:06 scripts

[ptomulik]

https://stackoverflow.com/a/18378785/2186051

[ptomulik]

Looks like it's related to interceptFileFuncs(). If I add Phar::interceptFileFuncs() at the beginning of my bin/test.php, then it works as expected (checking for files in real filesystem). EDIT: looks it's completely opposite (we need to remove all occurrences of interceptFileFuncs()).

#!/usr/bin/env php
<?php
Phar::interceptFileFuncs();

// ...

Another solution, I found, was to use box to create PHAR. Sami seemed to use box. For my bin/test.php I've used the following box.json configuration for box

{
  "output": "build/test.phar",
  "compactors": [
    "KevinGH\\Box\\Compactor\\Php"
  ],
  "main": "bin/test.php"
}

and just executed box compile. I guess, box prepends the Phar::interceptFileFuncs() call by default.

[williamdes]

Oh cool, I would just have to split or do some magic to adjust my script. Maybe one day migrate to a more fancy way of creating the phar, like box that I already starred

[williamdes]

I re-wrote the phar stub adding the line you asked me for, please test it and let me know Deployed as 5.3.0-dev

[ptomulik]

Still same error.

[williamdes]

Strange because I added the line: https://github.com/code-lts/doctum/blob/4fae085c573a3d75a0a0d059b125620dad506a50/scripts/phar-generator-script.php#L58

[ptomulik]

$ ./doctum.phar --version && ./doctum.phar update --force -v docs/doctum-relative.conf.php
Doctum 5.3.0-dev by Fabien Potencier and William Desportes
 Updating project

Version main
-------------

In Filesystem.php line 105:

  [Symfony\Component\Filesystem\Exception\IOException]
  Failed to create "docs/cache/html": mkdir(): File exists

Exception trace:
  at phar:///tmp/test/doctum.phar/vendor/symfony/filesystem/Filesystem.php:105
 Symfony\Component\Filesystem\Filesystem->mkdir() at phar:///tmp/test/doctum.phar/src/Project.php:380
 Doctum\Project->flushDir() at phar:///tmp/test/doctum.phar/src/Project.php:431
 Doctum\Project->prepareDir() at phar:///tmp/test/doctum.phar/src/Project.php:374
 Doctum\Project->getCacheDir() at phar:///tmp/test/doctum.phar/src/Store/JsonStore.php:76
 Doctum\Store\JsonStore->getStoreDir() at phar:///tmp/test/doctum.phar/src/Store/JsonStore.php:64
 Doctum\Store\JsonStore->flushProject() at phar:///tmp/test/doctum.phar/src/Project.php:464
 Doctum\Project->parseVersion() at phar:///tmp/test/doctum.phar/src/Project.php:125
 Doctum\Project->update() at phar:///tmp/test/doctum.phar/src/Console/Command/Command.php:177
 Doctum\Console\Command\Command->update() at phar:///tmp/test/doctum.phar/src/Console/Command/UpdateCommand.php:54
 Doctum\Console\Command\UpdateCommand->execute() at phar:///tmp/test/doctum.phar/vendor/symfony/console/Command/Command.php:255
 Symfony\Component\Console\Command\Command->run() at phar:///tmp/test/doctum.phar/vendor/symfony/console/Application.php:1009
 Symfony\Component\Console\Application->doRunCommand() at phar:///tmp/test/doctum.phar/vendor/symfony/console/Application.php:273
 Symfony\Component\Console\Application->doRun() at phar:///tmp/test/doctum.phar/vendor/symfony/console/Application.php:149
 Symfony\Component\Console\Application->run() at phar:///tmp/test/doctum.phar/bin/doctum-binary.php:26
 include() at /tmp/test/doctum.phar:17

update [--only-version ONLY-VERSION] [--force] [--output-format OUTPUT-FORMAT] [--no-progress] [--ignore-parse-errors] [--] <config>

[williamdes]

Well, is that an issue with phars or that the directory exists? I would love that you could help me debug this issue please

[ptomulik]

The issue is related to how certain filesystem-related functions work in phars, in this case is_dir() does not work as we expect. The interceptFileFuncs() should do the job, but it looks like it didn't. Maybe we should take a closer look, how and where it should be placed.

[williamdes]

Could you please check if the sami Phar does not work too? And create a new issue so we can solve this issue :)

[ptomulik]

Downloaded sami.phar and it works well with relative paths.

[ptomulik]

Seems I messed up things. It's something about Phar::interceptFileFuncs() but I wonder why, during my previous experiments I found I need it for is_dir() to work. Now it looks completely opposite - is_dir() doesn't work with relative paths when interceptFileFuncs() is present.

[williamdes]

Is this related: https://stackoverflow.com/a/48015791/5155484 ?

[ptomulik]

Is this related: https://stackoverflow.com/a/48015791/5155484 ?

Yes, this is related. Seems that #22 does the job finally. If interceptFileFuncs() is called, the file functions look for relative files within phar:// filesystem (but we want it to work on real filesystem). The default stub, doctum used previosuly, called interceptFileFuncs() internally as stated in docs for createDefaultStub. Now, after removing the line with interceptFileFuncs() everything seems to work as expected.

I still can't wrap my head around how did I came to completelly opposite conclusions previously (mean, that interceptFileFuncs() should be presend, while it appears, it should be actually absent).

[williamdes]

Thank you for researching about that, I really appreciate :)

[williamdes]

I released the 5.3.1-dev phar