search

craftcms / cms

Build bespoke content experiences with Craft.
https://craftcms.com
Other
3.28k stars 635 forks source link

PHP Exception when using S3 volumes and maxCachedCloudImageSize=0 #9884

Closed fleaz closed 3 years ago

fleaz commented 3 years ago

Description

Hey,

we run all our CraftCMS installations on AWS therefore we use S3 as our asset storage and deliver them directly via Cloudfront out of the bucket. Hence we don't need any of the images on the server and have set maxCachedCloudImageSize to 0 to never save any originals on the server itself so we don't have to provision them with a lot of storage. The problem resulting out of this setup is that we get A LOT of PHP exceptions from the getLocalImageSource() function in AssetTransforms.php because there is an unlink() at the beginning that always want's to delete the file but the file never exists in our setup so unlink throws an exception (which really hurts us because we use Sentry to track our exceptions and 99,5% of the related to this).

Let me know if you need anything else from me :)

Best, Felix

Steps to reproduce

  1. Use AWS S3 for a volume
  2. Set maxCachedCloudImageSize to 0 in your config
  3. Open a page for the first time so Craft has to generate a new transformation of an image

Additional info

andris-sevcenko commented 3 years ago

@fleaz What Craft version are you using? As of Craft 3.4.16, all of the exceptions should be caught.

fleaz commented 3 years ago

I just checked a few of our customers where we see this behavior and we run 3.4.21, 3.4.30 and 3.6.18

andris-sevcenko commented 3 years ago

Can you post the full stack trace, please?

fleaz commented 3 years ago

Yes of course. This is from the customer where we run 3.6.18 

ErrorException: Warning: unlink(/mnt/customername/htdocs/storage/runtime/assets/sources/2828586.jpg): No such file or directory
#14 /mnt/customername/htdocs/vendor/yiisoft/yii2/helpers/BaseFileHelper.php(416): yii\helpers\BaseFileHelper::unlink
#13 /mnt/customername/htdocs/vendor/craftcms/cms/src/helpers/FileHelper.php(439): craft\helpers\FileHelper::unlink
#12 /mnt/customername/htdocs/vendor/craftcms/cms/src/services/AssetTransforms.php(1064): craft\services\AssetTransforms::getLocalImageSource
#11 /mnt/customername/htdocs/vendor/craftcms/cms/src/services/Assets.php(749): craft\services\Assets::getThumbPath
#10 /mnt/customername/htdocs/vendor/craftcms/cms/src/controllers/AssetsController.php(1130): craft\controllers\AssetsController::actionThumb
#9 [internal](0): call_user_func_array
#8 /mnt/customername/htdocs/vendor/yiisoft/yii2/base/InlineAction.php(57): yii\base\InlineAction::runWithParams
#7 /mnt/customername/htdocs/vendor/yiisoft/yii2/base/Controller.php(181): yii\base\Controller::runAction
#6 /mnt/customername/htdocs/vendor/craftcms/cms/src/web/Controller.php(190): craft\web\Controller::runAction
#5 /mnt/customername/htdocs/vendor/yiisoft/yii2/base/Module.php(534): yii\base\Module::runAction
#4 /mnt/customername/htdocs/vendor/craftcms/cms/src/web/Application.php(278): craft\web\Application::runAction
#3 /mnt/customername/htdocs/vendor/craftcms/cms/src/web/Application.php(591): craft\web\Application::_processActionRequest
#2 /mnt/customername/htdocs/vendor/craftcms/cms/src/web/Application.php(257): craft\web\Application::handleRequest
#1 /mnt/customername/htdocs/vendor/yiisoft/yii2/base/Application.php(392): yii\base\Application::run
#0 /index.php(22): null
andris-sevcenko commented 3 years ago

It all sounds a bit suspicious. Can you, please, send over your composer.lock file as well as the following files:

I appreciate the relative sensitiveness of the composer.lock file, so feel free to email that to support@craftcms.com and reference this issue!

fleaz commented 3 years ago
/vendor/craftcms/cms/src/helpers/FileHelper.php ```php * @since 3.0.0 */ class FileHelper extends \yii\helpers\FileHelper { /** * @inheritdoc */ public static $mimeMagicFile = '@app/config/mimeTypes.php'; /** * @var bool Whether file locks can be used when writing to files. * @see useFileLocks() */ private static $_useFileLocks; /** * @inheritdoc */ public static function normalizePath($path, $ds = DIRECTORY_SEPARATOR) { // Is this a UNC network share path? $isUnc = (strpos($path, '//') === 0 || strpos($path, '\\\\') === 0); // Normalize the path $path = parent::normalizePath($path, $ds); // If it is UNC, add those slashes back in front if ($isUnc) { $path = $ds . $ds . ltrim($path, $ds); } return $path; } /** * @inheritdoc */ public static function copyDirectory($src, $dst, $options = []) { if (!isset($options['fileMode'])) { $options['fileMode'] = Craft::$app->getConfig()->getGeneral()->defaultFileMode; } if (!isset($options['dirMode'])) { $options['dirMode'] = Craft::$app->getConfig()->getGeneral()->defaultDirMode; } parent::copyDirectory($src, $dst, $options); } /** * @inheritdoc */ public static function createDirectory($path, $mode = null, $recursive = true) { if ($mode === null) { $mode = Craft::$app->getConfig()->getGeneral()->defaultDirMode; } return parent::createDirectory($path, $mode, $recursive); } /** * @inheritdoc */ public static function removeDirectory($dir, $options = []) { try { parent::removeDirectory($dir, $options); } catch (ErrorException $e) { // Try Symfony's thing as a fallback $fs = new Filesystem(); try { $fs->remove($dir); } catch (IOException $e2) { // throw the original exception instead throw $e; } } } /** * Sanitizes a filename. * * @param string $filename the filename to sanitize * @param array $options options for sanitization. Valid options are: * - `asciiOnly`: bool, whether only ASCII characters should be allowed. Defaults to false. * - `separator`: string|null, the separator character to use in place of whitespace. defaults to '-'. If set to null, whitespace will be preserved. * @return string The cleansed filename */ public static function sanitizeFilename(string $filename, array $options = []): string { $asciiOnly = $options['asciiOnly'] ?? false; $separator = array_key_exists('separator', $options) ? $options['separator'] : '-'; $disallowedChars = [ '—', '–', '‘', '’', '“', '”', '–', '—', '+', '%', '^', '~', '?', '[', ']', '/', '\\', '=', '<', '>', ':', ';', ',', '\'', '"', '&', '$', '#', '*', '(', ')', '|', '~', '`', '!', '{', '}', ]; // Replace any control characters in the name with a space. $filename = preg_replace("/\\x{00a0}/iu", ' ', $filename); // Strip any characters not allowed. $filename = str_replace($disallowedChars, '', strip_tags($filename)); if (Craft::$app->getDb()->getIsMysql()) { // Strip emojis $filename = StringHelper::replaceMb4($filename, ''); } // Nuke any trailing or leading .-_ $filename = trim($filename, '.-_'); $filename = $asciiOnly ? StringHelper::toAscii($filename) : $filename; if ($separator !== null) { $qSeparator = preg_quote($separator, '/'); $filename = preg_replace("/[\s{$qSeparator}]+/u", $separator, $filename); $filename = preg_replace("/^{$qSeparator}+|{$qSeparator}+$/u", '', $filename); } return $filename; } /** * Returns whether a given directory is empty (has no files) recursively. * * @param string $dir the directory to be checked * @return bool whether the directory is empty * @throws InvalidArgumentException if the dir is invalid * @throws ErrorException in case of failure */ public static function isDirectoryEmpty(string $dir): bool { if (!is_dir($dir)) { throw new InvalidArgumentException("The dir argument must be a directory: $dir"); } if (!($handle = opendir($dir))) { throw new ErrorException("Unable to open the directory: $dir"); } // It's empty until we find a file $empty = true; while (($file = readdir($handle)) !== false) { if ($file === '.' || $file === '..') { continue; } $path = $dir . DIRECTORY_SEPARATOR . $file; if (is_file($path) || !static::isDirectoryEmpty($path)) { $empty = false; break; } } closedir($handle); return $empty; } /** * Tests whether a file/directory is writable. * * @param string $path the file/directory path to test * @return bool whether the path is writable * @throws ErrorException in case of failure */ public static function isWritable(string $path): bool { // If it's a directory, test on a temp sub file if (is_dir($path)) { return static::isWritable($path . DIRECTORY_SEPARATOR . uniqid('test_writable', true) . '.tmp'); } // Remember whether the file already existed $exists = file_exists($path); if (($f = @fopen($path, 'ab')) === false) { return false; } @fclose($f); // Delete the file if it didn't exist already if (!$exists) { static::unlink($path); } return true; } /** * @inheritdoc */ public static function getMimeType($file, $magicFile = null, $checkExtension = true) { $mimeType = parent::getMimeType($file, $magicFile, $checkExtension); // Be forgiving of SVG files, etc., that don't have an XML declaration if ($checkExtension && ($mimeType === null || !static::canTrustMimeType($mimeType))) { return static::getMimeTypeByExtension($file, $magicFile) ?? $mimeType; } // Handle invalid SVG mime type reported by PHP (https://bugs.php.net/bug.php?id=79045) if (strpos($mimeType, 'image/svg') === 0) { return 'image/svg+xml'; } return $mimeType; } /** * Returns whether a MIME type can be trusted, or whether we should double-check based on the file extension. * * @param string $mimeType * @return bool * @since 3.1.7 */ public static function canTrustMimeType(string $mimeType): bool { return !in_array($mimeType, [ 'application/octet-stream', 'application/xml', 'text/html', 'text/plain', 'text/xml', ], true); } /** * Returns whether the given file path is an SVG image. * * @param string $file the file name. * @param string $magicFile name of the optional magic database file (or alias), usually something like `/path/to/magic.mime`. * This will be passed as the second parameter to [finfo_open()](http://php.net/manual/en/function.finfo-open.php) * when the `fileinfo` extension is installed. If the MIME type is being determined based via [[getMimeTypeByExtension()]] * and this is null, it will use the file specified by [[mimeMagicFile]]. * @param bool $checkExtension whether to use the file extension to determine the MIME type in case * `finfo_open()` cannot determine it. * @return bool */ public static function isSvg(string $file, ?string $magicFile = null, bool $checkExtension = true): bool { return self::getMimeType($file, $magicFile, $checkExtension) === 'image/svg+xml'; } /** * Returns whether the given file path is an GIF image. * * @param string $file the file name. * @param string $magicFile name of the optional magic database file (or alias), usually something like `/path/to/magic.mime`. * This will be passed as the second parameter to [finfo_open()](http://php.net/manual/en/function.finfo-open.php) * when the `fileinfo` extension is installed. If the MIME type is being determined based via [[getMimeTypeByExtension()]] * and this is null, it will use the file specified by [[mimeMagicFile]]. * @param bool $checkExtension whether to use the file extension to determine the MIME type in case * `finfo_open()` cannot determine it. * @return bool * @since 3.0.9 */ public static function isGif(string $file, ?string $magicFile = null, bool $checkExtension = true): bool { $mimeType = self::getMimeType($file, $magicFile, $checkExtension); return $mimeType === 'image/gif'; } /** * Writes contents to a file. * * @param string $file the file path * @param string $contents the new file contents * @param array $options options for file write. Valid options are: * - `createDirs`: bool, whether to create parent directories if they do * not exist. Defaults to `true`. * - `append`: bool, whether the contents should be appended to the * existing contents. Defaults to false. * - `lock`: bool, whether a file lock should be used. Defaults to the * `useWriteFileLock` config setting. * @throws InvalidArgumentException if the parent directory doesn't exist and `options[createDirs]` is `false` * @throws Exception if the parent directory can't be created * @throws ErrorException in case of failure */ public static function writeToFile(string $file, string $contents, array $options = []) { $file = static::normalizePath($file); $dir = dirname($file); if (!is_dir($dir)) { if (!isset($options['createDirs']) || $options['createDirs']) { static::createDirectory($dir); } else { throw new InvalidArgumentException("Cannot write to \"{$file}\" because the parent directory doesn't exist."); } } if (isset($options['lock'])) { $lock = (bool)$options['lock']; } else { $lock = static::useFileLocks(); } if ($lock) { $mutex = Craft::$app->getMutex(); $lockName = md5($file); if (!$mutex->acquire($lockName, 2)) { throw new ErrorException("Unable to acquire a lock for file \"{$file}\"."); } } else { $lockName = $mutex = null; } $flags = 0; if (!empty($options['append'])) { $flags |= FILE_APPEND; } if (file_put_contents($file, $contents, $flags) === false) { throw new ErrorException("Unable to write new contents to \"{$file}\"."); } // Invalidate opcache static::invalidate($file); if ($lock) { $mutex->release($lockName); } } /** * Creates a `.gitignore` file in the given directory if one doesn’t exist yet. * * @param string $path * @param array $options options for file write. Valid options are: * - `createDirs`: bool, whether to create parent directories if they do * not exist. Defaults to `true`. * - `lock`: bool, whether a file lock should be used. Defaults to `false`. * @throws InvalidArgumentException if the parent directory doesn't exist and `options[createDirs]` is `false` * @throws Exception if the parent directory can't be created * @throws ErrorException in case of failure * @since 3.4.0 */ public static function writeGitignoreFile(string $path, array $options = []) { $gitignorePath = $path . DIRECTORY_SEPARATOR . '.gitignore'; if (is_file($gitignorePath)) { return; } $contents = "*\n!.gitignore\n"; $options = array_merge([ // Prevent a segfault if this is called recursively 'lock' => false, ], $options); static::writeToFile($gitignorePath, $contents, $options); } /** * Removes a file or symlink in a cross-platform way * * @param string $path the file to be deleted * @return bool * @deprecated in 3.0.0-RC11. Use [[unlink()]] instead. */ public static function removeFile(string $path): bool { return static::unlink($path); } /** * @inheritdoc * @since 3.4.16 */ public static function unlink($path) { // BaseFileHelper::unlink() doesn't seem to catch all possible exceptions try { return parent::unlink($path); } catch (\Throwable $e) { return false; } } /** * Removes all of a directory’s contents recursively. * * @param string $dir the directory to be deleted recursively. * @param array $options options for directory remove. Valid options are: * - `traverseSymlinks`: bool, whether symlinks to the directories should be traversed too. * Defaults to `false`, meaning the content of the symlinked directory would not be deleted. * Only symlink would be removed in that default case. * - `filter`: callback (see [[findFiles()]]) * - `except`: array (see [[findFiles()]]) * - `only`: array (see [[findFiles()]]) * @throws InvalidArgumentException if the dir is invalid * @throws ErrorException in case of failure */ public static function clearDirectory(string $dir, array $options = []) { if (!is_dir($dir)) { throw new InvalidArgumentException("The dir argument must be a directory: $dir"); } // Adapted from [[removeDirectory()]], plus addition of filters, and minus the root directory removal at the end if (!($handle = opendir($dir))) { return; } if (!isset($options['basePath'])) { $options['basePath'] = realpath($dir); $options = static::normalizeOptions($options); } while (($file = readdir($handle)) !== false) { if ($file === '.' || $file === '..') { continue; } $path = $dir . DIRECTORY_SEPARATOR . $file; if (static::filterPath($path, $options)) { if (is_dir($path)) { static::removeDirectory($path, $options); } else { static::unlink($path); } } } closedir($handle); } /** * Returns the last modification time for the given path. * * If the path is a directory, any nested files/directories will be checked as well. * * @param string $path the directory to be checked * @return int Unix timestamp representing the last modification time */ public static function lastModifiedTime($path) { if (is_file($path)) { return filemtime($path); } $times = [filemtime($path)]; $iterator = new RecursiveIteratorIterator( new RecursiveDirectoryIterator($path, FilesystemIterator::SKIP_DOTS), RecursiveIteratorIterator::SELF_FIRST); foreach ($iterator as $p => $info) { $times[] = filemtime($p); } return max($times); } /** * Returns whether any files in a source directory have changed, compared to another directory. * * @param string $dir the source directory to check for changes in * @param string $ref the reference directory * @return bool * @throws InvalidArgumentException if $dir or $ref isn't a directory * @throws ErrorException if we can't get a handle on $src */ public static function hasAnythingChanged(string $dir, string $ref): bool { if (!is_dir($dir)) { throw new InvalidArgumentException("The src argument must be a directory: {$dir}"); } if (!is_dir($ref)) { throw new InvalidArgumentException("The ref argument must be a directory: {$ref}"); } if (!($handle = opendir($dir))) { throw new ErrorException("Unable to open the directory: {$dir}"); } while (($file = readdir($handle)) !== false) { if ($file === '.' || $file === '..') { continue; } $path = $dir . DIRECTORY_SEPARATOR . $file; $refPath = $ref . DIRECTORY_SEPARATOR . $file; if (is_dir($path)) { if (!is_dir($refPath) || static::hasAnythingChanged($path, $refPath)) { return true; } } else if (!is_file($refPath) || filemtime($path) > filemtime($refPath)) { return true; } } return false; } /** * Returns whether file locks can be used when writing to files. * * @return bool */ public static function useFileLocks(): bool { if (self::$_useFileLocks !== null) { return self::$_useFileLocks; } $generalConfig = Craft::$app->getConfig()->getGeneral(); if (is_bool($generalConfig->useFileLocks)) { return self::$_useFileLocks = $generalConfig->useFileLocks; } // Do we have it cached? $cacheService = Craft::$app->getCache(); if (($cachedVal = $cacheService->get('useFileLocks')) !== false) { return self::$_useFileLocks = ($cachedVal === 'y'); } // Try a test lock self::$_useFileLocks = false; try { $mutex = Craft::$app->getMutex(); $name = uniqid('test_lock', true); if (!$mutex->acquire($name)) { throw new Exception('Unable to acquire test lock.'); } if (!$mutex->release($name)) { throw new Exception('Unable to release test lock.'); } self::$_useFileLocks = true; } catch (\Throwable $e) { Craft::warning('Write lock test failed: ' . $e->getMessage(), __METHOD__); } // Cache for two months $cachedValue = self::$_useFileLocks ? 'y' : 'n'; $cacheService->set('useFileLocks', $cachedValue, 5184000); return self::$_useFileLocks; } /** * Moves existing files down to `*.1`, `*.2`, etc. * * @param string $basePath The base path to the first file (sans `.X`) * @param int $max The most files that can coexist before we should start deleting them * @since 3.0.38 */ public static function cycle(string $basePath, int $max = 50) { // Go through all of them and move them forward. for ($i = $max; $i > 0; $i--) { $thisFile = $basePath . ($i == 1 ? '' : '.' . ($i - 1)); if (file_exists($thisFile)) { if ($i === $max) { @unlink($thisFile); } else { @rename($thisFile, "$basePath.$i"); } } } } /** * Invalidates a cached file with `clearstatcache()` and `opcache_invalidate()`. * * @param string $file the file path * @since 3.4.0 */ public static function invalidate(string $file) { clearstatcache(true, $file); if (function_exists('opcache_invalidate') && filter_var(ini_get('opcache.enable'), FILTER_VALIDATE_BOOLEAN)) { @opcache_invalidate($file, true); } } /** * Zips a file. * * @param string $path the file/directory path * @param string|null $to the target zip file path. If null, the original path will be used, with “.zip” appended to it. * @return string the zip file path * @throws InvalidArgumentException if `$path` is not a valid file/directory path * @throws Exception if the zip cannot be created * @since 3.5.0 */ public static function zip(string $path, ?string $to = null): string { $path = static::normalizePath($path); if (!file_exists($path)) { throw new InvalidArgumentException("No file/directory exists at $path"); } if ($to === null) { $to = "$path.zip"; } $zip = new ZipArchive(); if ($zip->open($to, ZipArchive::CREATE) !== true) { throw new Exception("Cannot create zip at $to"); } $name = basename($path); if (is_file($path)) { $zip->addFile($path, $name); } else { static::addFilesToZip($zip, $path); } $zip->close(); return $to; } /** * Adds all the files in a given directory to a ZipArchive, preserving the nested directory structure. * * @param ZipArchive $zip the ZipArchive object * @param string $dir the directory path * @param string|null $prefix the path prefix to use when adding the contents of the directory * @param array $options options for file searching. See [[findFiles()]] for available options. * @param 3.5.0 */ public static function addFilesToZip(ZipArchive $zip, string $dir, ?string $prefix = null, $options = []) { if (!is_dir($dir)) { return; } if ($prefix !== null) { $prefix = static::normalizePath($prefix) . '/'; } else { $prefix = ''; } $files = static::findFiles($dir, $options); foreach ($files as $file) { // Use forward slashes $file = str_replace(DIRECTORY_SEPARATOR, '/', $file); // Preserve the directory structure within the templates folder $zip->addFile($file, $prefix . substr($file, strlen($dir) + 1)); } } /** * Return a file extension for the given MIME type. * * @param $mimeType * @return string * @throws InvalidArgumentException if no known extensions exist for the given MIME type. * @since 3.5.15 */ public static function getExtensionByMimeType($mimeType): string { $extensions = FileHelper::getExtensionsByMimeType($mimeType); if (empty($extensions)) { throw new InvalidArgumentException("No file extensions are known for the MIME Type $mimeType."); } $extension = reset($extensions); // Manually correct for some types. switch ($extension) { case 'svgz': return 'svg'; case 'jpe': return 'jpg'; } return $extension; } } ```
/vendor/yiisoft/yii2/helpers/BaseFileHelper.php ```php * @author Alex Makarov * @since 2.0 */ class BaseFileHelper { const PATTERN_NODIR = 1; const PATTERN_ENDSWITH = 4; const PATTERN_MUSTBEDIR = 8; const PATTERN_NEGATIVE = 16; const PATTERN_CASE_INSENSITIVE = 32; /** * @var string the path (or alias) of a PHP file containing MIME type information. */ public static $mimeMagicFile = '@yii/helpers/mimeTypes.php'; /** * @var string the path (or alias) of a PHP file containing MIME aliases. * @since 2.0.14 */ public static $mimeAliasesFile = '@yii/helpers/mimeAliases.php'; /** * Normalizes a file/directory path. * * The normalization does the following work: * * - Convert all directory separators into `DIRECTORY_SEPARATOR` (e.g. "\a/b\c" becomes "/a/b/c") * - Remove trailing directory separators (e.g. "/a/b/c/" becomes "/a/b/c") * - Turn multiple consecutive slashes into a single one (e.g. "/a///b/c" becomes "/a/b/c") * - Remove ".." and "." based on their meanings (e.g. "/a/./b/../c" becomes "/a/c") * * Note: For registered stream wrappers, the consecutive slashes rule * and ".."/"." translations are skipped. * * @param string $path the file/directory path to be normalized * @param string $ds the directory separator to be used in the normalized result. Defaults to `DIRECTORY_SEPARATOR`. * @return string the normalized file/directory path */ public static function normalizePath($path, $ds = DIRECTORY_SEPARATOR) { $path = rtrim(strtr($path, '/\\', $ds . $ds), $ds); if (strpos($ds . $path, "{$ds}.") === false && strpos($path, "{$ds}{$ds}") === false) { return $path; } // fix #17235 stream wrappers foreach (stream_get_wrappers() as $protocol) { if (strpos($path, "{$protocol}://") === 0) { return $path; } } // the path may contain ".", ".." or double slashes, need to clean them up if (strpos($path, "{$ds}{$ds}") === 0 && $ds == '\\') { $parts = [$ds]; } else { $parts = []; } foreach (explode($ds, $path) as $part) { if ($part === '..' && !empty($parts) && end($parts) !== '..') { array_pop($parts); } elseif ($part === '.' || $part === '' && !empty($parts)) { continue; } else { $parts[] = $part; } } $path = implode($ds, $parts); return $path === '' ? '.' : $path; } /** * Returns the localized version of a specified file. * * The searching is based on the specified language code. In particular, * a file with the same name will be looked for under the subdirectory * whose name is the same as the language code. For example, given the file "path/to/view.php" * and language code "zh-CN", the localized file will be looked for as * "path/to/zh-CN/view.php". If the file is not found, it will try a fallback with just a language code that is * "zh" i.e. "path/to/zh/view.php". If it is not found as well the original file will be returned. * * If the target and the source language codes are the same, * the original file will be returned. * * @param string $file the original file * @param string $language the target language that the file should be localized to. * If not set, the value of [[\yii\base\Application::language]] will be used. * @param string $sourceLanguage the language that the original file is in. * If not set, the value of [[\yii\base\Application::sourceLanguage]] will be used. * @return string the matching localized file, or the original file if the localized version is not found. * If the target and the source language codes are the same, the original file will be returned. */ public static function localize($file, $language = null, $sourceLanguage = null) { if ($language === null) { $language = Yii::$app->language; } if ($sourceLanguage === null) { $sourceLanguage = Yii::$app->sourceLanguage; } if ($language === $sourceLanguage) { return $file; } $desiredFile = dirname($file) . DIRECTORY_SEPARATOR . $language . DIRECTORY_SEPARATOR . basename($file); if (is_file($desiredFile)) { return $desiredFile; } $language = substr($language, 0, 2); if ($language === $sourceLanguage) { return $file; } $desiredFile = dirname($file) . DIRECTORY_SEPARATOR . $language . DIRECTORY_SEPARATOR . basename($file); return is_file($desiredFile) ? $desiredFile : $file; } /** * Determines the MIME type of the specified file. * This method will first try to determine the MIME type based on * [finfo_open](https://secure.php.net/manual/en/function.finfo-open.php). If the `fileinfo` extension is not installed, * it will fall back to [[getMimeTypeByExtension()]] when `$checkExtension` is true. * @param string $file the file name. * @param string $magicFile name of the optional magic database file (or alias), usually something like `/path/to/magic.mime`. * This will be passed as the second parameter to [finfo_open()](https://secure.php.net/manual/en/function.finfo-open.php) * when the `fileinfo` extension is installed. If the MIME type is being determined based via [[getMimeTypeByExtension()]] * and this is null, it will use the file specified by [[mimeMagicFile]]. * @param bool $checkExtension whether to use the file extension to determine the MIME type in case * `finfo_open()` cannot determine it. * @return string|null the MIME type (e.g. `text/plain`). Null is returned if the MIME type cannot be determined. * @throws InvalidConfigException when the `fileinfo` PHP extension is not installed and `$checkExtension` is `false`. */ public static function getMimeType($file, $magicFile = null, $checkExtension = true) { if ($magicFile !== null) { $magicFile = Yii::getAlias($magicFile); } if (!extension_loaded('fileinfo')) { if ($checkExtension) { return static::getMimeTypeByExtension($file, $magicFile); } throw new InvalidConfigException('The fileinfo PHP extension is not installed.'); } $info = finfo_open(FILEINFO_MIME_TYPE, $magicFile); if ($info) { $result = finfo_file($info, $file); finfo_close($info); if ($result !== false) { return $result; } } return $checkExtension ? static::getMimeTypeByExtension($file, $magicFile) : null; } /** * Determines the MIME type based on the extension name of the specified file. * This method will use a local map between extension names and MIME types. * @param string $file the file name. * @param string $magicFile the path (or alias) of the file that contains all available MIME type information. * If this is not set, the file specified by [[mimeMagicFile]] will be used. * @return string|null the MIME type. Null is returned if the MIME type cannot be determined. */ public static function getMimeTypeByExtension($file, $magicFile = null) { $mimeTypes = static::loadMimeTypes($magicFile); if (($ext = pathinfo($file, PATHINFO_EXTENSION)) !== '') { $ext = strtolower($ext); if (isset($mimeTypes[$ext])) { return $mimeTypes[$ext]; } } return null; } /** * Determines the extensions by given MIME type. * This method will use a local map between extension names and MIME types. * @param string $mimeType file MIME type. * @param string $magicFile the path (or alias) of the file that contains all available MIME type information. * If this is not set, the file specified by [[mimeMagicFile]] will be used. * @return array the extensions corresponding to the specified MIME type */ public static function getExtensionsByMimeType($mimeType, $magicFile = null) { $aliases = static::loadMimeAliases(static::$mimeAliasesFile); if (isset($aliases[$mimeType])) { $mimeType = $aliases[$mimeType]; } $mimeTypes = static::loadMimeTypes($magicFile); return array_keys($mimeTypes, mb_strtolower($mimeType, 'UTF-8'), true); } private static $_mimeTypes = []; /** * Loads MIME types from the specified file. * @param string $magicFile the path (or alias) of the file that contains all available MIME type information. * If this is not set, the file specified by [[mimeMagicFile]] will be used. * @return array the mapping from file extensions to MIME types */ protected static function loadMimeTypes($magicFile) { if ($magicFile === null) { $magicFile = static::$mimeMagicFile; } $magicFile = Yii::getAlias($magicFile); if (!isset(self::$_mimeTypes[$magicFile])) { self::$_mimeTypes[$magicFile] = require $magicFile; } return self::$_mimeTypes[$magicFile]; } private static $_mimeAliases = []; /** * Loads MIME aliases from the specified file. * @param string $aliasesFile the path (or alias) of the file that contains MIME type aliases. * If this is not set, the file specified by [[mimeAliasesFile]] will be used. * @return array the mapping from file extensions to MIME types * @since 2.0.14 */ protected static function loadMimeAliases($aliasesFile) { if ($aliasesFile === null) { $aliasesFile = static::$mimeAliasesFile; } $aliasesFile = Yii::getAlias($aliasesFile); if (!isset(self::$_mimeAliases[$aliasesFile])) { self::$_mimeAliases[$aliasesFile] = require $aliasesFile; } return self::$_mimeAliases[$aliasesFile]; } /** * Copies a whole directory as another one. * The files and sub-directories will also be copied over. * @param string $src the source directory * @param string $dst the destination directory * @param array $options options for directory copy. Valid options are: * * - dirMode: integer, the permission to be set for newly copied directories. Defaults to 0775. * - fileMode: integer, the permission to be set for newly copied files. Defaults to the current environment setting. * - filter: callback, a PHP callback that is called for each directory or file. * The signature of the callback should be: `function ($path)`, where `$path` refers the full path to be filtered. * The callback can return one of the following values: * * * true: the directory or file will be copied (the "only" and "except" options will be ignored) * * false: the directory or file will NOT be copied (the "only" and "except" options will be ignored) * * null: the "only" and "except" options will determine whether the directory or file should be copied * * - only: array, list of patterns that the file paths should match if they want to be copied. * A path matches a pattern if it contains the pattern string at its end. * For example, '.php' matches all file paths ending with '.php'. * Note, the '/' characters in a pattern matches both '/' and '\' in the paths. * If a file path matches a pattern in both "only" and "except", it will NOT be copied. * - except: array, list of patterns that the files or directories should match if they want to be excluded from being copied. * A path matches a pattern if it contains the pattern string at its end. * Patterns ending with '/' apply to directory paths only, and patterns not ending with '/' * apply to file paths only. For example, '/a/b' matches all file paths ending with '/a/b'; * and '.svn/' matches directory paths ending with '.svn'. Note, the '/' characters in a pattern matches * both '/' and '\' in the paths. * - caseSensitive: boolean, whether patterns specified at "only" or "except" should be case sensitive. Defaults to true. * - recursive: boolean, whether the files under the subdirectories should also be copied. Defaults to true. * - beforeCopy: callback, a PHP callback that is called before copying each sub-directory or file. * If the callback returns false, the copy operation for the sub-directory or file will be cancelled. * The signature of the callback should be: `function ($from, $to)`, where `$from` is the sub-directory or * file to be copied from, while `$to` is the copy target. * - afterCopy: callback, a PHP callback that is called after each sub-directory or file is successfully copied. * The signature of the callback should be: `function ($from, $to)`, where `$from` is the sub-directory or * file copied from, while `$to` is the copy target. * - copyEmptyDirectories: boolean, whether to copy empty directories. Set this to false to avoid creating directories * that do not contain files. This affects directories that do not contain files initially as well as directories that * do not contain files at the target destination because files have been filtered via `only` or `except`. * Defaults to true. This option is available since version 2.0.12. Before 2.0.12 empty directories are always copied. * @throws InvalidArgumentException if unable to open directory */ public static function copyDirectory($src, $dst, $options = []) { $src = static::normalizePath($src); $dst = static::normalizePath($dst); if ($src === $dst || strpos($dst, $src . DIRECTORY_SEPARATOR) === 0) { throw new InvalidArgumentException('Trying to copy a directory to itself or a subdirectory.'); } $dstExists = is_dir($dst); if (!$dstExists && (!isset($options['copyEmptyDirectories']) || $options['copyEmptyDirectories'])) { static::createDirectory($dst, isset($options['dirMode']) ? $options['dirMode'] : 0775, true); $dstExists = true; } $handle = opendir($src); if ($handle === false) { throw new InvalidArgumentException("Unable to open directory: $src"); } if (!isset($options['basePath'])) { // this should be done only once $options['basePath'] = realpath($src); $options = static::normalizeOptions($options); } while (($file = readdir($handle)) !== false) { if ($file === '.' || $file === '..') { continue; } $from = $src . DIRECTORY_SEPARATOR . $file; $to = $dst . DIRECTORY_SEPARATOR . $file; if (static::filterPath($from, $options)) { if (isset($options['beforeCopy']) && !call_user_func($options['beforeCopy'], $from, $to)) { continue; } if (is_file($from)) { if (!$dstExists) { // delay creation of destination directory until the first file is copied to avoid creating empty directories static::createDirectory($dst, isset($options['dirMode']) ? $options['dirMode'] : 0775, true); $dstExists = true; } copy($from, $to); if (isset($options['fileMode'])) { @chmod($to, $options['fileMode']); } } else { // recursive copy, defaults to true if (!isset($options['recursive']) || $options['recursive']) { static::copyDirectory($from, $to, $options); } } if (isset($options['afterCopy'])) { call_user_func($options['afterCopy'], $from, $to); } } } closedir($handle); } /** * Removes a directory (and all its content) recursively. * * @param string $dir the directory to be deleted recursively. * @param array $options options for directory remove. Valid options are: * * - traverseSymlinks: boolean, whether symlinks to the directories should be traversed too. * Defaults to `false`, meaning the content of the symlinked directory would not be deleted. * Only symlink would be removed in that default case. * * @throws ErrorException in case of failure */ public static function removeDirectory($dir, $options = []) { if (!is_dir($dir)) { return; } if (!empty($options['traverseSymlinks']) || !is_link($dir)) { if (!($handle = opendir($dir))) { return; } while (($file = readdir($handle)) !== false) { if ($file === '.' || $file === '..') { continue; } $path = $dir . DIRECTORY_SEPARATOR . $file; if (is_dir($path)) { static::removeDirectory($path, $options); } else { static::unlink($path); } } closedir($handle); } if (is_link($dir)) { static::unlink($dir); } else { rmdir($dir); } } /** * Removes a file or symlink in a cross-platform way * * @param string $path * @return bool * * @since 2.0.14 */ public static function unlink($path) { $isWindows = DIRECTORY_SEPARATOR === '\\'; if (!$isWindows) { return unlink($path); } if (is_link($path) && is_dir($path)) { return rmdir($path); } try { return unlink($path); } catch (ErrorException $e) { // last resort measure for Windows if (is_dir($path) && count(static::findFiles($path)) !== 0) { return false; } if (function_exists('exec') && file_exists($path)) { exec('DEL /F/Q ' . escapeshellarg($path)); return !file_exists($path); } return false; } } /** * Returns the files found under the specified directory and subdirectories. * @param string $dir the directory under which the files will be looked for. * @param array $options options for file searching. Valid options are: * * - `filter`: callback, a PHP callback that is called for each directory or file. * The signature of the callback should be: `function ($path)`, where `$path` refers the full path to be filtered. * The callback can return one of the following values: * * * `true`: the directory or file will be returned (the `only` and `except` options will be ignored) * * `false`: the directory or file will NOT be returned (the `only` and `except` options will be ignored) * * `null`: the `only` and `except` options will determine whether the directory or file should be returned * * - `except`: array, list of patterns excluding from the results matching file or directory paths. * Patterns ending with slash ('/') apply to directory paths only, and patterns not ending with '/' * apply to file paths only. For example, '/a/b' matches all file paths ending with '/a/b'; * and `.svn/` matches directory paths ending with `.svn`. * If the pattern does not contain a slash (`/`), it is treated as a shell glob pattern * and checked for a match against the pathname relative to `$dir`. * Otherwise, the pattern is treated as a shell glob suitable for consumption by `fnmatch(3)` * with the `FNM_PATHNAME` flag: wildcards in the pattern will not match a `/` in the pathname. * For example, `views/*.php` matches `views/index.php` but not `views/controller/index.php`. * A leading slash matches the beginning of the pathname. For example, `/*.php` matches `index.php` but not `views/start/index.php`. * An optional prefix `!` which negates the pattern; any matching file excluded by a previous pattern will become included again. * If a negated pattern matches, this will override lower precedence patterns sources. Put a backslash (`\`) in front of the first `!` * for patterns that begin with a literal `!`, for example, `\!important!.txt`. * Note, the '/' characters in a pattern matches both '/' and '\' in the paths. * - `only`: array, list of patterns that the file paths should match if they are to be returned. Directory paths * are not checked against them. Same pattern matching rules as in the `except` option are used. * If a file path matches a pattern in both `only` and `except`, it will NOT be returned. * - `caseSensitive`: boolean, whether patterns specified at `only` or `except` should be case sensitive. Defaults to `true`. * - `recursive`: boolean, whether the files under the subdirectories should also be looked for. Defaults to `true`. * @return array files found under the directory, in no particular order. Ordering depends on the files system used. * @throws InvalidArgumentException if the dir is invalid. */ public static function findFiles($dir, $options = []) { $dir = self::clearDir($dir); $options = self::setBasePath($dir, $options); $list = []; $handle = self::openDir($dir); while (($file = readdir($handle)) !== false) { if ($file === '.' || $file === '..') { continue; } $path = $dir . DIRECTORY_SEPARATOR . $file; if (static::filterPath($path, $options)) { if (is_file($path)) { $list[] = $path; } elseif (is_dir($path) && (!isset($options['recursive']) || $options['recursive'])) { $list = array_merge($list, static::findFiles($path, $options)); } } } closedir($handle); return $list; } /** * Returns the directories found under the specified directory and subdirectories. * @param string $dir the directory under which the files will be looked for. * @param array $options options for directory searching. Valid options are: * * - `filter`: callback, a PHP callback that is called for each directory or file. * The signature of the callback should be: `function ($path)`, where `$path` refers the full path to be filtered. * The callback can return one of the following values: * * * `true`: the directory will be returned * * `false`: the directory will NOT be returned * * - `recursive`: boolean, whether the files under the subdirectories should also be looked for. Defaults to `true`. * @return array directories found under the directory, in no particular order. Ordering depends on the files system used. * @throws InvalidArgumentException if the dir is invalid. * @since 2.0.14 */ public static function findDirectories($dir, $options = []) { $dir = self::clearDir($dir); $options = self::setBasePath($dir, $options); $list = []; $handle = self::openDir($dir); while (($file = readdir($handle)) !== false) { if ($file === '.' || $file === '..') { continue; } $path = $dir . DIRECTORY_SEPARATOR . $file; if (is_dir($path) && static::filterPath($path, $options)) { $list[] = $path; if (!isset($options['recursive']) || $options['recursive']) { $list = array_merge($list, static::findDirectories($path, $options)); } } } closedir($handle); return $list; } /** * @param string $dir */ private static function setBasePath($dir, $options) { if (!isset($options['basePath'])) { // this should be done only once $options['basePath'] = realpath($dir); $options = static::normalizeOptions($options); } return $options; } /** * @param string $dir */ private static function openDir($dir) { $handle = opendir($dir); if ($handle === false) { throw new InvalidArgumentException("Unable to open directory: $dir"); } return $handle; } /** * @param string $dir */ private static function clearDir($dir) { if (!is_dir($dir)) { throw new InvalidArgumentException("The dir argument must be a directory: $dir"); } return rtrim($dir, DIRECTORY_SEPARATOR); } /** * Checks if the given file path satisfies the filtering options. * @param string $path the path of the file or directory to be checked * @param array $options the filtering options. See [[findFiles()]] for explanations of * the supported options. * @return bool whether the file or directory satisfies the filtering options. */ public static function filterPath($path, $options) { if (isset($options['filter'])) { $result = call_user_func($options['filter'], $path); if (is_bool($result)) { return $result; } } if (empty($options['except']) && empty($options['only'])) { return true; } $path = str_replace('\\', '/', $path); if (!empty($options['except'])) { if (($except = self::lastExcludeMatchingFromList($options['basePath'], $path, $options['except'])) !== null) { return $except['flags'] & self::PATTERN_NEGATIVE; } } if (!empty($options['only']) && !is_dir($path)) { if (($except = self::lastExcludeMatchingFromList($options['basePath'], $path, $options['only'])) !== null) { // don't check PATTERN_NEGATIVE since those entries are not prefixed with ! return true; } return false; } return true; } /** * Creates a new directory. * * This method is similar to the PHP `mkdir()` function except that * it uses `chmod()` to set the permission of the created directory * in order to avoid the impact of the `umask` setting. * * @param string $path path of the directory to be created. * @param int $mode the permission to be set for the created directory. * @param bool $recursive whether to create parent directories if they do not exist. * @return bool whether the directory is created successfully * @throws \yii\base\Exception if the directory could not be created (i.e. php error due to parallel changes) */ public static function createDirectory($path, $mode = 0775, $recursive = true) { if (is_dir($path)) { return true; } $parentDir = dirname($path); // recurse if parent dir does not exist and we are not at the root of the file system. if ($recursive && !is_dir($parentDir) && $parentDir !== $path) { static::createDirectory($parentDir, $mode, true); } try { if (!mkdir($path, $mode)) { return false; } } catch (\Exception $e) { if (!is_dir($path)) {// https://github.com/yiisoft/yii2/issues/9288 throw new \yii\base\Exception("Failed to create directory \"$path\": " . $e->getMessage(), $e->getCode(), $e); } } try { return chmod($path, $mode); } catch (\Exception $e) { throw new \yii\base\Exception("Failed to change permissions for directory \"$path\": " . $e->getMessage(), $e->getCode(), $e); } } /** * Performs a simple comparison of file or directory names. * * Based on match_basename() from dir.c of git 1.8.5.3 sources. * * @param string $baseName file or directory name to compare with the pattern * @param string $pattern the pattern that $baseName will be compared against * @param int|bool $firstWildcard location of first wildcard character in the $pattern * @param int $flags pattern flags * @return bool whether the name matches against pattern */ private static function matchBasename($baseName, $pattern, $firstWildcard, $flags) { if ($firstWildcard === false) { if ($pattern === $baseName) { return true; } } elseif ($flags & self::PATTERN_ENDSWITH) { /* "*literal" matching against "fooliteral" */ $n = StringHelper::byteLength($pattern); if (StringHelper::byteSubstr($pattern, 1, $n) === StringHelper::byteSubstr($baseName, -$n, $n)) { return true; } } $matchOptions = []; if ($flags & self::PATTERN_CASE_INSENSITIVE) { $matchOptions['caseSensitive'] = false; } return StringHelper::matchWildcard($pattern, $baseName, $matchOptions); } /** * Compares a path part against a pattern with optional wildcards. * * Based on match_pathname() from dir.c of git 1.8.5.3 sources. * * @param string $path full path to compare * @param string $basePath base of path that will not be compared * @param string $pattern the pattern that path part will be compared against * @param int|bool $firstWildcard location of first wildcard character in the $pattern * @param int $flags pattern flags * @return bool whether the path part matches against pattern */ private static function matchPathname($path, $basePath, $pattern, $firstWildcard, $flags) { // match with FNM_PATHNAME; the pattern has base implicitly in front of it. if (strpos($pattern, '/') === 0) { $pattern = StringHelper::byteSubstr($pattern, 1, StringHelper::byteLength($pattern)); if ($firstWildcard !== false && $firstWildcard !== 0) { $firstWildcard--; } } $namelen = StringHelper::byteLength($path) - (empty($basePath) ? 0 : StringHelper::byteLength($basePath) + 1); $name = StringHelper::byteSubstr($path, -$namelen, $namelen); if ($firstWildcard !== 0) { if ($firstWildcard === false) { $firstWildcard = StringHelper::byteLength($pattern); } // if the non-wildcard part is longer than the remaining pathname, surely it cannot match. if ($firstWildcard > $namelen) { return false; } if (strncmp($pattern, $name, $firstWildcard)) { return false; } $pattern = StringHelper::byteSubstr($pattern, $firstWildcard, StringHelper::byteLength($pattern)); $name = StringHelper::byteSubstr($name, $firstWildcard, $namelen); // If the whole pattern did not have a wildcard, then our prefix match is all we need; we do not need to call fnmatch at all. if (empty($pattern) && empty($name)) { return true; } } $matchOptions = [ 'filePath' => true ]; if ($flags & self::PATTERN_CASE_INSENSITIVE) { $matchOptions['caseSensitive'] = false; } return StringHelper::matchWildcard($pattern, $name, $matchOptions); } /** * Scan the given exclude list in reverse to see whether pathname * should be ignored. The first match (i.e. the last on the list), if * any, determines the fate. Returns the element which * matched, or null for undecided. * * Based on last_exclude_matching_from_list() from dir.c of git 1.8.5.3 sources. * * @param string $basePath * @param string $path * @param array $excludes list of patterns to match $path against * @return array|null null or one of $excludes item as an array with keys: 'pattern', 'flags' * @throws InvalidArgumentException if any of the exclude patterns is not a string or an array with keys: pattern, flags, firstWildcard. */ private static function lastExcludeMatchingFromList($basePath, $path, $excludes) { foreach (array_reverse($excludes) as $exclude) { if (is_string($exclude)) { $exclude = self::parseExcludePattern($exclude, false); } if (!isset($exclude['pattern']) || !isset($exclude['flags']) || !isset($exclude['firstWildcard'])) { throw new InvalidArgumentException('If exclude/include pattern is an array it must contain the pattern, flags and firstWildcard keys.'); } if ($exclude['flags'] & self::PATTERN_MUSTBEDIR && !is_dir($path)) { continue; } if ($exclude['flags'] & self::PATTERN_NODIR) { if (self::matchBasename(basename($path), $exclude['pattern'], $exclude['firstWildcard'], $exclude['flags'])) { return $exclude; } continue; } if (self::matchPathname($path, $basePath, $exclude['pattern'], $exclude['firstWildcard'], $exclude['flags'])) { return $exclude; } } return null; } /** * Processes the pattern, stripping special characters like / and ! from the beginning and settings flags instead. * @param string $pattern * @param bool $caseSensitive * @return array with keys: (string) pattern, (int) flags, (int|bool) firstWildcard * @throws InvalidArgumentException */ private static function parseExcludePattern($pattern, $caseSensitive) { if (!is_string($pattern)) { throw new InvalidArgumentException('Exclude/include pattern must be a string.'); } $result = [ 'pattern' => $pattern, 'flags' => 0, 'firstWildcard' => false, ]; if (!$caseSensitive) { $result['flags'] |= self::PATTERN_CASE_INSENSITIVE; } if (empty($pattern)) { return $result; } if (strpos($pattern, '!') === 0) { $result['flags'] |= self::PATTERN_NEGATIVE; $pattern = StringHelper::byteSubstr($pattern, 1, StringHelper::byteLength($pattern)); } if (StringHelper::byteLength($pattern) && StringHelper::byteSubstr($pattern, -1, 1) === '/') { $pattern = StringHelper::byteSubstr($pattern, 0, -1); $result['flags'] |= self::PATTERN_MUSTBEDIR; } if (strpos($pattern, '/') === false) { $result['flags'] |= self::PATTERN_NODIR; } $result['firstWildcard'] = self::firstWildcardInPattern($pattern); if (strpos($pattern, '*') === 0 && self::firstWildcardInPattern(StringHelper::byteSubstr($pattern, 1, StringHelper::byteLength($pattern))) === false) { $result['flags'] |= self::PATTERN_ENDSWITH; } $result['pattern'] = $pattern; return $result; } /** * Searches for the first wildcard character in the pattern. * @param string $pattern the pattern to search in * @return int|bool position of first wildcard character or false if not found */ private static function firstWildcardInPattern($pattern) { $wildcards = ['*', '?', '[', '\\']; $wildcardSearch = function ($r, $c) use ($pattern) { $p = strpos($pattern, $c); return $r === false ? $p : ($p === false ? $r : min($r, $p)); }; return array_reduce($wildcards, $wildcardSearch, false); } /** * @param array $options raw options * @return array normalized options * @since 2.0.12 */ protected static function normalizeOptions(array $options) { if (!array_key_exists('caseSensitive', $options)) { $options['caseSensitive'] = true; } if (isset($options['except'])) { foreach ($options['except'] as $key => $value) { if (is_string($value)) { $options['except'][$key] = self::parseExcludePattern($value, $options['caseSensitive']); } } } if (isset($options['only'])) { foreach ($options['only'] as $key => $value) { if (is_string($value)) { $options['only'][$key] = self::parseExcludePattern($value, $options['caseSensitive']); } } } return $options; } } ```

The lockfile is on it's way via email.

Thanks for looking into this! :)

andris-sevcenko commented 3 years ago

Alright, I just fixed this for the next release. This hadn't come to our attention, because the exception thrown by Yii is caught by Craft. With the Sentry plugin, however, it intercepts the exception before Craft can see it and decide to do nothing about it.

From now on, Craft will just make sure a file exists before trying to delete it.

brandonkelly commented 3 years ago

Craft 3.7.15 is out now with the fix for this.

fleaz commented 3 years ago

Awesome! Thank you both for fixing this :)