touch(): CLI Cache Flush Permission Issue

[dev-rke]

Hi,

first i'd like to thank you for this very helpful extension - it saved us a lot of time in development.

We are using typo3_console to deploy our TYPO3 applications. To deploy the application we use a special user "deploy" which is in the primary group "www-data". The folders are owned by the deploy user. When we trigger cache:flush via typo3_console, unfortunately the application fails to deploy, because of the asset cache of the vhs extension.

Unfortunately the vhs extensions uses "touch" to reset the timestamp of their temp files, when the cache is flushed. In this case the user permissions are required to be the same as the files are.

I solved this by myself by giving the vhs-assets files the permission "deploy:www-data". But when somebody removes the typo3temp folder, the permission issue comes back.

A collegue of mine solved it by overwriting the AssetService with an own class:

foreach ($assetCacheFiles as $assetCacheFile) {
            // START FIX to update file modification/creation time ;)
            $content = file_get_contents($assetCacheFile);
            unlink($assetCacheFile);
            $this->writeFile($assetCacheFile, $content);
            // END FIX to update file modification/creation time ;)

            touch($assetCacheFile, 0);
}

This works, because any user can unlink any files which are not write protected and create files with his own rights. You can reproduce this issue by creating a new Linux user "deploy" with primary group "www-data" and executing the following commands:

# create new user
$ useradd -G www-data deploy

$ cd typo3temp/assets/

# set permissions as the webserver would do
$ sudo chmod www-data:www-data vhs-assets-*

# try to touch
$ sudo -u deploy touch -t 197001010000 vhs-assets-*.js

Result:

touch: setting times of 'vhs-assets-8a17dbbe899569ad0ea77a739ce58992.js': Operation not permitted

In my opinion the source of the issue is the utime system call:

Changing timestamps is permitted when: either the process has appropri‐ ate privileges, or the effective user ID equals the user ID of the file, or times is NULL and the process has write permission for the file.

See this for more details: https://unix.stackexchange.com/questions/64848/cannot-touch-m-a-writable-file

I am not sure if the issue is to be called a bug, but it is an issue which increases complexity of deployments. One solution would be to reset file ownership before clearing caches. Another solution would be to not re-use the typo3temp folder, just remove it and create it from scratch, before running the cache:flush command. A third approach would be to exclude the vhs caches from beeing flushed when using CLI.

Therefore i would appreciate to discuss some ideas to find out a best practice to solve this issue.

To be complete, here is also the stacktrace of typo3_console:

$ TYPO3_CONTEXT=Production/Cli /usr/bin/php /var/www/releases/current/vendor/bin/typo3cms cache:flush

[ Helhum\Typo3Console\Mvc\Cli\FailedSubProcessCommandException ]                      
#1485130941: Executing command "cache:flushcomplete" failed (exit code: "1")          
thrown in file vendor/helhum/typo3-console/Classes/Command/CacheCommandController.php 
in line 76                                                                            

Command line:
'/usr/bin/php7.0' '/var/www/releases/current/vendor/bin/typo3cms' 'cache:flushcomplete'

Exception trace:
#0 Helhum\Typo3Console\Mvc\Cli\CommandDispatcher::executeCommand()
  vendor/helhum/typo3-console/Classes/Command/CacheCommandController.php:76
#1 Helhum\Typo3Console\Command\CacheCommandController::flushCommand()
#2 call_user_func_array()
  vendor/helhum/typo3-console/Classes/Mvc/Controller/CommandController.php:252
#3 Helhum\Typo3Console\Mvc\Controller\CommandController::callCommandMethod()
  vendor/helhum/typo3-console/Classes/Mvc/Controller/CommandController.php:137
#4 Helhum\Typo3Console\Mvc\Controller\CommandController::processRequest()
  typo3/sysext/extbase/Classes/Mvc/Dispatcher.php:85
#5 TYPO3\CMS\Extbase\Mvc\Dispatcher::dispatch()
  vendor/helhum/typo3-console/Classes/Mvc/Cli/RequestHandler.php:80
#6 Helhum\Typo3Console\Mvc\Cli\RequestHandler::handleRequest()
  vendor/helhum/typo3-console/Classes/Core/ConsoleBootstrap.php:111
#7 Helhum\Typo3Console\Core\ConsoleBootstrap::run()
  vendor/helhum/typo3-console/Scripts/typo3cms.php:60
#8 {closure}()
  vendor/helhum/typo3-console/Scripts/typo3cms.php:61
#9 require()
  vendor/helhum/typo3-console/Scripts/typo3cms:4

[ Sub-process exception: TYPO3\CMS\Core\Error\Exception ]                                                                                                                        
#1: Warning: touch(): Utime failed: Operation not permitted in /var/www/releases/current/web/typo3conf/ext/vhs/Classes/Service/AssetService.php line 755 
thrown in file vendor/helhum/typo3-console/Classes/Error/ErrorHandler.php                                                                                                        
in line 68                                                                                                                                                                       

Exception trace:
#0 Helhum\Typo3Console\Error\ErrorHandler::handleError()
#1 touch()
  typo3conf/ext/vhs/Classes/Service/AssetService.php:755
#2 FluidTYPO3\Vhs\Service\AssetService::clearCacheCommand()
#3 call_user_func_array()
  typo3/sysext/core/Classes/Utility/GeneralUtility.php:3833
#4 TYPO3\CMS\Core\Utility\GeneralUtility::callUserFunction()
  typo3/sysext/core/Classes/DataHandling/DataHandler.php:8735
#5 TYPO3\CMS\Core\DataHandling\DataHandler::clear_cacheCmd()
  vendor/helhum/typo3-console/Classes/Service/CacheService.php:117
#6 Helhum\Typo3Console\Service\CacheService::flushCachesWithDataHandler()
  vendor/helhum/typo3-console/Classes/Command/CacheCommandController.php:110
#7 Helhum\Typo3Console\Command\CacheCommandController::flushCompleteCommand()
#8 call_user_func_array()
  vendor/helhum/typo3-console/Classes/Mvc/Controller/CommandController.php:252
#9 Helhum\Typo3Console\Mvc\Controller\CommandController::callCommandMethod()
  vendor/helhum/typo3-console/Classes/Mvc/Controller/CommandController.php:137
#10 Helhum\Typo3Console\Mvc\Controller\CommandController::processRequest()
  typo3/sysext/extbase/Classes/Mvc/Dispatcher.php:85
#11 TYPO3\CMS\Extbase\Mvc\Dispatcher::dispatch()
  vendor/helhum/typo3-console/Classes/Mvc/Cli/RequestHandler.php:80
#12 Helhum\Typo3Console\Mvc\Cli\RequestHandler::handleRequest()
  vendor/helhum/typo3-console/Classes/Core/ConsoleBootstrap.php:111
#13 Helhum\Typo3Console\Core\ConsoleBootstrap::run()
  vendor/helhum/typo3-console/Scripts/typo3cms.php:60
#14 {closure}()
  vendor/helhum/typo3-console/Scripts/typo3cms.php:61
#15 require()
  vendor/helhum/typo3-console/Scripts/typo3cms:4

[mkarulin]

Good to see that i am not alone with this problem :D Still, no reply?

[NamelessCoder]

The touch function call is used to expire files without physically removing them which would break frontend requests. You are completely right about the owner/group concerns though, which more and more come into the scene when we start doing deployment with different users - which of course should be possible.

We would accept a patch for this as pull request, using the solution proposed by @dev-rke, but with one modification: the file should be written first to a different filename then moved to overwrite the old file, since that removes any possibility that a frontend request might 404 in precisely the microsecond or three it takes to write a normal-sized asset file.

@mkarulin

Still, no reply?

I'm keeping pretty busy with a project that you know and can only take a couple of issues now and then ;)

[jonakieling]

Hi everyone,

we stumbled upon this issue too. We tried the solution above and found another issue. We are allowed to operate on the VHS caches via CLI but afterwards the webserver user has the same issue for the same reasons. Leading to exception when clearing the cache from the Backend.

Our two cents after some research and fidling around:

We now do our deployments as the web server user by setting the uid and gid of our deployment user. Our hosting service already does this on a per-website-basis for all ssh users anyway.

Setting a user's uid and gid must be done directly in /etc/passwd as root or with sudo.

Hopefully this helps a bit. We understand that it is very setup-specific.

Kind regards, Jona

[h-e-l-l-o-w-o-r-l-d]

+1

We have this issue, too.