search

hydephp / framework

The HydePHP Framework
https://hydephp.com
MIT License
13 stars 2 forks source link

BuildService finding non-existent files to copy in Debian #662

Open clifmo opened 2 weeks ago

clifmo commented 2 weeks ago

I'm using Docker to build and I just sort of expected this to work. I've tried multiple version of PHP, different Debian releases. In short, BuildService tries to copy a file named _media.css which is the name of the folder, and not one of the file contents.

I started using Alpine but PHP glob is a known issue. So it's interesting that this also relates to glob(). MediaFile.php:109

I haven't had a chance to debug it but I'm curious why the directory is being considered as an asset, and what I can do to avoid this. Composer wants ^8.1 but I get a composer platform check error on 8.1. What am I missing?


    protected static function getMediaAssetFiles(): array
    {
        return glob(Hyde::path(static::getMediaGlobPattern()), GLOB_BRACE) ?: [];
    }

    protected static function getMediaGlobPattern(): string
    {
        return sprintf(Hyde::getMediaDirectory().'/{*,**/*,**/*/*}.{%s}', implode(',',
            Config::getArray('hyde.media_extensions', self::EXTENSIONS)
        ));
    }
FROM php:8.2-cli-buster AS build-php

WORKDIR  /app

COPY . .

RUN php hyde build
> [4/4] RUN php hyde build:
0.143
0.143
0.143                           Building your static site!
0.143
0.143
0.147 Removing all files from build directory...
0.147
0.147 Transferring Media Assets...
0.152  0/1 [░░░░░░░░░░░░░░░░░░░░░░░░░░░░]   0%
0.156    ErrorException
0.156
0.156   copy(/app/_media/_media.css): Failed to open stream: No such file or directory
0.156
0.156   at vendor/hyde/framework/src/Framework/Services/BuildService.php:60
0.157      56▕         $this->comment('Transferring Media Assets...');
0.157      57▕         $this->withProgressBar(MediaFile::files(), function (string $identifier): void {
0.157      58▕             $sitePath = Hyde::siteMediaPath($identifier);
0.157      59▕             $this->needsParentDirectory($sitePath);
0.157   ➜  60▕             copy(Hyde::mediaPath($identifier), $sitePath);
0.157      61▕         });
0.157      62▕
0.157      63▕         $this->newLine(2);
0.157      64▕     }
0.157
0.157       +17 vendor frames
0.157
0.157   18  hyde:35
0.157       LaravelZero\Framework\Kernel::handle(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
0.157
------
Dockerfile:8
--------------------
   6 |
   7 |     #RUN php hyde publish:homepage posts -n
   8 | >>> RUN php hyde build
caendesilva commented 2 weeks ago

This feel very weird. Could you provide a full ls/tree of the source media directory so I can reproduce?

@clifmo Are you using _media.css as the filename of the directory? I could see that being the cause of the bug, since the glob doesn't differentiate. Should be a fairly easy fix, would just want the ls -r output of _media so I can verify a fix works for your problem.

clifmo commented 2 weeks ago

Sure I saw this on a fresh install and my active install only has one PNG file and app CSS. I'll provide reproduction steps once I'm at my PC

clifmo commented 2 weeks ago

I have CI/CD working in my homelab via Gitea Actions, a private Docker registry, and Watchtower to pull in updates. So I build via Dockerfile and publish in my action. Was hoping this could work but glob has been trouble.

Install composer create-project hyde/hyde hyde-test && cd hyde-test

Check Everthing looks good. php hyde info php hyde serve

ls -r _media
app.css

Docker Time

echo "FROM php:8.2-cli-buster AS build-php

WORKDIR  /app

COPY . .

RUN php hyde build" >> Dockerfile

docker build .

 > [4/4] RUN php hyde build:
0.188
0.188
0.188                           Building your static site!
0.188
0.188
0.190 Removing all files from build directory...
0.190
0.190 Transferring Media Assets...
0.193  0/1 [░░░░░░░░░░░░░░░░░░░░░░░░░░░░]   0%
0.195    ErrorException
0.195
0.195   copy(/app/_media/_media.css): Failed to open stream: No such file or directory
0.195
0.195   at vendor/hyde/framework/src/Framework/Services/BuildService.php:60
0.195      56▕         $this->comment('Transferring Media Assets...');
0.195      57▕         $this->withProgressBar(MediaFile::files(), function (string $identifier): void {
0.195      58▕             $sitePath = Hyde::siteMediaPath($identifier);
0.195      59▕             $this->needsParentDirectory($sitePath);
0.195   ➜  60▕             copy(Hyde::mediaPath($identifier), $sitePath);
0.195      61▕         });
0.195      62▕
0.195      63▕         $this->newLine(2);
0.195      64▕     }
0.195
0.195       +17 vendor frames
0.195
0.195   18  hyde:35
0.195       LaravelZero\Framework\Kernel::handle(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
0.195
------
Dockerfile:7
--------------------
   5 |     COPY . .
   6 |
   7 | >>> RUN php hyde build
   8 |
--------------------
ERROR: failed to solve: process "/bin/sh -c php hyde build" did not complete successfully: exit code: 1

Build from host: macOS Sequoia 15.0.1

❯ php hyde build

                          Building your static site!

Removing all files from build directory...

Transferring Media Assets...
 1/1 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100%

Creating Blade Pages...
 2/2 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100%

All done! Finished in 0.02 seconds (21.02ms) with 11.79MB peak memory usage
Congratulations! 🎉 Your static site has been built!
Your new homepage is stored here -> [..]/hyde-test/_site/index.html
❯ ls -r _media
app.css
❯ php -v
PHP 8.3.13 (cli) (built: Oct 22 2024 18:39:14) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.3.13, Copyright (c) Zend Technologies
    with Zend OPcache v8.3.13, Copyright (c), by Zend Technologies

FROM php:8.3 AS build-php is the same result.

caendesilva commented 2 weeks ago

Thank you so much for the detailed reproduction steps! I'll get right on debugging this.

caendesilva commented 2 weeks ago

It feels like glob() is the cause of many cross-platform issues (for example not existing on , so am considering switching to RecursiveIteratorIterator for improved stability in a future update.

I believe this issue is due to Debian having a different implementation for handling glob patterns. I tried a patch in https://github.com/hydephp/framework/commit/802ad4fa6aeb05cd338fec419d01a22682711219, could you try to update your Docker environment to try out this branch in order to test if the fix works for you?

You can run this command to try out the development branch:

composer config repositories.framework vcs https://github.com/hydephp/framework && composer require "hyde/framework:dev-check-media-file-directories" --update-with-dependencies
clifmo commented 2 weeks ago

Well, I was working off my host, then copying into a docker environment during docker build. I did this entirely inside the container and didn't have issues.

I still had the same issue with my Dockerfile. I did a few more steps to prove it.

Dockerfile

Commented the RUN build command.

FROM php:8.2-cli-buster AS build-php

WORKDIR  /app

COPY . .

# RUN php hyde build

Build

❯ docker build . -f Dockerfile.test -t temp:latest
[+] Building 1.5s (8/8) FINISHED                                                                                                                                                   docker:desktop-linux
 => [internal] load build definition from Dockerfile.test                                                                                                                                          0.0s
 => => transferring dockerfile: 126B                                                                                                                                                               0.0s
 => [internal] load metadata for docker.io/library/php:8.2-cli-buster                                                                                                                              0.8s
 => [internal] load .dockerignore                                                                                                                                                                  0.0s
 => => transferring context: 2B                                                                                                                                                                    0.0s
 => [1/3] FROM docker.io/library/php:8.2-cli-buster@sha256:c1a06439f7b38487a79e0eebe5ccfea1f86978f68b6d51bbba3482c38622fa8d                                                                        0.0s
 => [internal] load build context                                                                                                                                                                  0.2s
 => => transferring context: 391.39kB                                                                                                                                                              0.2s
 => CACHED [2/3] WORKDIR  /app                                                                                                                                                                     0.0s
 => [3/3] COPY . .                                                                                                                                                                                 0.3s
 => exporting to image                                                                                                                                                                             0.1s
 => => exporting layers                                                                                                                                                                            0.1s
 => => writing image sha256:43eadbdad5488dc8265d1726b7e73522d2bcc38cd4dd48fd2dc198d571adec7a                                                                                                       0.0s
 => => naming to docker.io/library/temp:latest

Run into built container

docker run -it temp:latest sh

Check _media

# ls -r _media
app.css

Ensure dev framework

# cat vendor/hyde/framework/src/Support/Filesystem/MediaFile.php | grep "array_filter($files"
        return array_filter($files, 'is_file');

Build

# php hyde build

                          Building your static site!

Removing all files from build directory...

Transferring Media Assets...
 0/1 [░░░░░░░░░░░░░░░░░░░░░░░░░░░░]   0%
   ErrorException

  copy(/app/_media/_media.css): Failed to open stream: No such file or directory

  at vendor/hyde/framework/src/Framework/Services/BuildService.php:60
     56▕         $this->comment('Transferring Media Assets...');
     57▕         $this->withProgressBar(MediaFile::files(), function (string $identifier): void {
     58▕             $sitePath = Hyde::siteMediaPath($identifier);
     59▕             $this->needsParentDirectory($sitePath);
  ➜  60▕             copy(Hyde::mediaPath($identifier), $sitePath);
     61▕         });
     62▕
     63▕         $this->newLine(2);
     64▕     }

      +17 vendor frames

  18  hyde:35
      LaravelZero\Framework\Kernel::handle(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

Docker Run

Fresh container without copying from host.

Host: docker run -it php:8.3-cli sh

Container:

php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');"
php -r "if (hash_file('sha384', 'composer-setup.php') === 'dac665fdc30fdd8ec78b38b9800061b4150413ff2e3b6f88543c636f7cd84f6db9189d43a81e5503cda447da73c7e5b6') { echo 'Installer verified'; } else { echo 'Installer corrupt'; unlink('composer-setup.php'); } echo PHP_EOL;"
php composer-setup.php
php -r "unlink('composer-setup.php');"# Installer verified

mv composer.phar /usr/local/bin/composer

apt update && apt install git composer create-project hyde/hyde hyde-test && cd hyde-test

# php hyde build

                          Building your static site!

Removing all files from build directory...

Transferring Media Assets...
 1/1 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100%

Creating Blade Pages...
 2/2 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100%

All done! Finished in 0.01 seconds (8.80ms) with 12.32MB peak memory usage
Congratulations! 🎉 Your static site has been built!
Your new homepage is stored here -> file:///hyde-test/_site/index.html
caendesilva commented 2 weeks ago

@clifmo Does it seem like your issue was fixed by the patch/proper Docker setup?

clifmo commented 2 weeks ago

@caendesilva It does not, unfortunately. I really do appreciate the help. In short, this only happens when copying data from the host system into the container.

I stuck a dd() to see what it sees during the build. There's only one result before the array_filter. It's in discoverMediaAssetFiles() ... tracking this into ProjectFile::construct() but taken it about as far as dd() will take me.

0.130 Transferring Media Assets...
0.132 array:1 [
0.132   0 => "/app/_media/app.css"
0.132 ]
    protected static function discoverMediaAssetFiles(): array
    {

        return collect(static::getMediaAssetFiles())->mapWithKeys(function (string $path): array {
            $file = static::make($path);
dd($file);
            return [$file->getIdentifier() => $file];
        })->all();
    }

 => ERROR [4/4] RUN php hyde build                                                                                                                                                                 0.1s
------
 > [4/4] RUN php hyde build:
0.117
0.117
0.117                           Building your static site!
0.117
0.117
0.119 Removing all files from build directory...
0.119
0.119 Transferring Media Assets...
0.121 Hyde\Support\Filesystem\MediaFile {#360
0.121   +path: "_media.css"
0.121 }
------
Dockerfile:8
--------------------
caendesilva commented 2 weeks ago

@clifmo Thanks for providing all these information points. Sadly, I am personally not knowledgeable enough in Docker to fully understand what quirks are happening here, so I asked on Twitter to see if anyone else could take a look so we can get this fixed.

In the meantime I will be working on switching to use the RecursiveDirectoryIterator as I think it will be more reliable, but I'm going on two out of state trips this week so it may take longer than I would wish for that to be released as I want to make sure it is done properly in order to not introduce any new issues.

clifmo commented 2 weeks ago

Have safe trips! I will possibly better learn Github Actions or hack something quick and dirty. Cheers

caendesilva commented 2 weeks ago

Have safe trips! I will possibly better learn Github Actions or hack something quick and dirty. Cheers

Thank you! By the way, we do have a GitHub Action if you're looking to build (and deploy) your Hyde sites from a CI. https://github.com/hydephp/action

Here is also a blog post if you want to make a more lower-level action using basic Action tooling. https://hydephp.com/posts/github-actions-deployment

Let me know if you need any help!

clifmo commented 2 weeks ago

Yea! I saw that. As mentioned, I'm deploying to a homelab so my go-to method is monitoring Docker image updates with Watchtower. I'll have to figure out something to grab the artifact and stick it into the bind mount path. I suppose I can just scp it since it's all within my network, but I'll have to dig deeper into the docker networking, since it's all on the same host.

caendesilva commented 2 weeks ago

@clifmo I got some answers on Twitter. Let me know if this could narrow it down for you.

either: they haven't created the missing fill outside the container to be copied, they did but at the wrong path when translated into the container, they didn't build the file inside the container if it's a compiled file, or something is slightly off between how their paths map https://x.com/MtgDank/status/1855746481937936658

Could you also confirm the following:

clifmo commented 2 weeks ago

I provided reproduction steps based on a clean install so there are no additional files. I built the simplest docker image possible. It's in the reproduction steps. I figured out an actions-based approach which required a bit more than docs suggest, at least for my setup in Gitea. Happy to contribute that back. Sorry to be short, it's Monday.

caendesilva commented 2 weeks ago

I provided reproduction steps based on a clean install so there are no additional files. I built the simplest docker image possible. It's in the reproduction steps. I figured out an actions-based approach which required a bit more than docs suggest, at least for my setup in Gitea. Happy to contribute that back. Sorry to be short, it's Monday.

Thanks for the information. Just wanted to double check that everything in the reproduction steps were complete as I am not too familiar with Docker.

I do have this Dockerfile which may be helpful, but I'm not 100% sure it works:

FROM php:8.1-cli

# Install system dependencies
RUN apt update && apt install zip unzip

# Install Composer
RUN curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer

# Download base project files 
RUN curl -L https://github.com/hydephp/hyde/archive/refs/heads/master.zip -o hyde.zip \
    && unzip hyde.zip \
    && rm hyde.zip \
    && mv hyde-master hyde

# Install Hyde
RUN cd /hyde && composer install

# Set the working directory
WORKDIR /hyde

# Make the HydeCLI executable
RUN chmod +x /hyde/hyde

# Set HydeCLI as the entrypoint
ENTRYPOINT ["/hyde/hyde"]

# Set the default command
CMD ["build"]

There is also an experimental official Docker image for the standalone CLI project that you could maybe use: https://github.com/hydephp/cli/

clifmo commented 2 weeks ago

Sounds good. I have moved on from this so there's no immediacy. However there still is something funky with the collection of media assets, and it's not glob. Although glob has its own issues. I will write up how I got actions working in a private network, Gitea, docker environment. Fair?

caendesilva commented 2 weeks ago

Sounds good. I have moved on from this so there's no immediacy. However there still is something funky with the collection of media assets, and it's not glob. Although glob has its own issues. I will write up how I got actions working in a private network, Gitea, docker environment. Fair?

That sounds great! Would love the read. I'll keep the issue open for a while, and work on moving away from glob, regardless of if it is relevant to this. Hope this doesn't discourage you from exploring HydePHP further.

clifmo commented 2 weeks ago

Absolutely not, you have been incredibly responsive. Next step is using the Laravel backend notifications to cross-post to socials. Has anyone you know worked on this? Staring with AT Protocol and ActivityPub

caendesilva commented 2 weeks ago

Absolutely not, you have been incredibly responsive. Next step is using the Laravel backend notifications to cross-post to socials. Has anyone you know worked on this? Staring with AT Protocol and ActivityPub

Glad to hear it, thank you so much! I have actually not tried using notifications from Hyde, I love the idea and is something I'd also love to read a writeup on. I have not tried AT Protocol or ActivityPub but any Laravel notification driver should work.

I think the build tasks feature would be great if you want to send notifications during the site build. https://hydephp.com/docs/1.x/build-tasks

clifmo commented 2 weeks ago

Definitely going to try. As promised, from idea to entirely self-hosted ci/CD in a weekend. https://blog.clifmo.com/posts/using-gitea-actions-to-deploy-hydephp-in-a-self-hosted-docker-environment.html

caendesilva commented 2 weeks ago

Definitely going to try. As promised, from idea to entirely self-hosted ci/CD in a weekend. https://blog.clifmo.com/posts/using-gitea-actions-to-deploy-hydephp-in-a-self-hosted-docker-environment.html

Amazing writeup! And thanks for the shoutout ❤️