[question] "lasso:pull --force" command for lasso:pull

StanBarrows commented 4 years ago

Hey @Sammyjo20

First of all, awesome stuff you're building here!

Today we've spent some time in the company experimenting with Lasso. We use Laravel Forge & Envoyer in combination with Github Actions to deploy our applications. For storing our assets, we've decided to use S3.

Laravel Envoyer is creating a new release for each deployment. So even if there is a history within your lasso folder we need every time a fresh pull from the S3 Server. Because of this reason we thought we can ignore the lasso-bundle.json.

If there is no history.json within the lasso folder the lasso:pull is working without any problems. But if there is an existing history.json in the file we receive the following error.

An exception was thrown. Reason: A valid "lasso-bundle.json" file could not be found in the Filesystem for the current environment.

Maybe we missing something but is there a way to provide something like a php artisan lasso:pull --force command, to receive always just the latest version? With ignoring the exiting lasso-bundle.json?

As a quick-fix: We've added the initial lasso-bundle.json back to the public folder.

Cheers Sebastian

Sammyjo20 commented 4 years ago

Hi Stan,

Thank you for using Lasso, I appreciate your kind words,

Absolutely we can add something like that in. However I just wanted to ask a couple of questions just in case it's a bug.

The pull command reads from the lasso-bundle.json, and not the history.json. It firstly looks to see if there's a lasso-bundle.json file within the repository, if there is - it will use that. If there isn't one in there (e.g you are using the --no-git flag), it will download the lasso-bundle.json file from the Filesystem and read the latest version from there.

The history.json file is only used to keep track of the previous bundles and it's used to delete the oldest after pulling.

I hope that helps explain things a little more - are you using Git or have you tried using the --no-git flag?

https://github.com/Sammyjo20/Lasso#publish

StanBarrows commented 4 years ago

Hi @Sammyjo20

Thanks for the quick response.

We've .gitignored

.lasso
lasso-bundle.json

Within our Github Actions we use the following step.

Receive the current GitHub branch name to define the LASSO_ENV

      - name: Set output
        id: vars
        run: echo ::set-output name=short_ref::${GITHUB_REF#refs/*/}

      - name: publish assets
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          AWS_DEFAULT_REGION: ${{ secrets.AWS_DEFAULT_REGION }}
          AWS_BUCKET: ${{ secrets.AWS_BUCKET }}
          AWS_URL: ${{ secrets.AWS_URL }}
          LASSO_ENV: ${{ steps.vars.outputs.short_ref }}
        run: php artisan lasso:publish --no-git

This stores our assets within the following structure.

On Envoyer we've setup the following build step.

cd {{release}} 

php artisan lasso:pull

Currently this throws the following error on the developer branch deployment on Envoyer.

An exception was thrown. Reason: A valid "lasso-bundle.json" file could not be found in the Filesystem for the current environment.

More Information https://flareapp.io/share/Lm8RNA7v#F16

Cheers Sebastian

StanBarrows commented 4 years ago

HI @Sammyjo20

I started debugging it on my local machine.

Somehow the lasso-bundle is not found on the Cloud This dd() on $this->cloud->exists($cloudPath) is returning false.

The Cloud Path Variable is "lasso/www.projectname.ch/developer/lasso-bundle.json" which is matching the structure on the S3 Bucket.

If i go a step further and try to get $file = $this->cloud->get($cloudPath), it crashes with ah Illuminate\Contracts\Filesystem\FileNotFoundException .

Cheers Sebastian

StanBarrows commented 4 years ago

Hi @Sammyjo20

I guess we found the problem...

I slightly changed the AWS Root Path after the Lasso Setup. I guess this was mixing up the LASSO_ENV and LASSO_PATH where we specified this again.

     's3' => [
            'driver' => 's3',
            'key' => env('AWS_ACCESS_KEY_ID'),
            'secret' => env('AWS_SECRET_ACCESS_KEY'),
            'region' => env('AWS_DEFAULT_REGION'),
            'bucket' => env('AWS_BUCKET'),
            'url' => env('AWS_URL'),
            'visibility' => 'public',
            'root' => env('AWS_PATH', 'default') . '/' .  env('APP_ENV', ''),
        ],

Problem is fixed and everything is running smooth!

So thanks for the help!

Cheers, Sebastian

Sammyjo20 commented 4 years ago

Glad you got it fixed Stan! Nicely done. Is Lasso working well for you now? Do you have any feedback/suggestions?

StanBarrows commented 4 years ago

Hey @Sammyjo20

Thanks for your message!

We just started using it today, but we like the concept of the whole package. It is a perfect fit between having your public assets stored within git, A complex CDN or generating your assets on the production server. Especially for smaller projects a welcome extension!

The most challenging thing for us was to figure out the complete setup & integration with S3 and Github Actions. There we found ourself sometimes a little lost within the existing documentation. So maybe an example setup page within the wiki for each service provider and perhaps how to implement it with CI/CD Tools like Github Actions would be an improvement.

I'm happy to share our current setup

Laravel Forge & Envoyer
Laravel 8
Yarn
Amazon S3
Github Actions

Install Composer Packages

composer require sammyjo20/lasso
composer require league/flysystem-aws-s3-v3
composer require league/flysystem-cached-adapter

Publish Lasso config

php artisan vendor:publish --tag=lasso-config

Modify the lasso config

We're using Yarn for the asset generation so we switched the script to

'script' => 'yarn build:prod',

We've excluded assets we would like to keep within our repository.

   'excluded_directories' => [
            'docs',
            'favicon'
        ],

switched storage.disk to S3

'disk' => 's3',

Adjust filesystem s3 Settings

Visibility You can either make ALL uploaded files in Laravel public or private. Without visibility set to public, you won't have URL access to your assets.

set all files public

     's3' => [
           //set all uploads to public visible
           'visibility' => 'public',
        ],

set only specific files public $path = Storage::disk('s3')->put(YOUR_PATH, $request->file('YOUR_FILENAME'), 'public');

Root-Path If you store other projects within your bucket, you could specify the root path within a new AWS_PATH variable and extend it with your current APP_ENV. With the following approach, you can ignore the built-in LASSO_ENV functionality.

     's3' => [
           //extend 'visibility'
            'root' => env('AWS_PATH') . '/' .  env('APP_ENV', ''),
        ],

AWS-Setup

Login to your management console | https://aws.amazon.com/console
Go To Services S3 | https://s3.console.aws.amazon.com
Create a new Bucket
Go To Identity and Access Management (IAM)
Create a new User with the existing AmazonS3FullAccess Permission
Save and update your .env variables

AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_DEFAULT_REGION=eu-central-1
AWS_BUCKET=
AWS_PATH=project_path

You can easily test your S3 Credentials via Transmit. Use s3.amazonaws.com as a Host url.

Github Actions

We've created an assets_deploy.yml Workflow within our .github/workflows folder.

name: assets & deploy

on:
  push:
    branches:
      - developer
      - staging
      - production

jobs:
  assets:
    runs-on: ubuntu-latest
    name: assets
    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Setup PHP
        uses: shivammathur/setup-php@v2
        with:
          php-version: '7.4'
          extensions: dom, curl, libxml, mbstring, zip, pcntl, pdo, sqlite, pdo_sqlite, bcmath, soap, intl, gd, exif, iconv, imagick

      - name: Install Composer dependencies
        run: composer install --prefer-dist --no-interaction --no-suggest

      - name: Prepare the environment
        run: cp .env.github_action .env

      - name: Generate app key
        run: php artisan key:generate

      - name: Install assets
        run: yarn install

      - name: Set output
        id: vars
        run: echo ::set-output name=short_ref::${GITHUB_REF#refs/*/}

      - name: publish assets
        env:
          APP_ENV: ${{ steps.vars.outputs.short_ref }}
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          AWS_DEFAULT_REGION: ${{ secrets.AWS_DEFAULT_REGION }}
          AWS_BUCKET: ${{ secrets.AWS_BUCKET }}
        run: php artisan lasso:publish --no-git

  deploy:
    needs: assets
    runs-on: ubuntu-latest
    name: deploy
    steps:
      - uses: actions/checkout@master
        with:
          fetch-depth: '0'
      - name: Deploy to developer
        env:
          ENVOYER_HOOK_DEVELOPER: ${{ secrets.ENVOYER_HOOK_DEVELOPER }}
        if: github.ref == 'refs/heads/developer'
        run: curl "$ENVOYER_HOOK_DEVELOPER"?sha=${{ github.sha }}

      - name: Deploy to staging
        env:
          ENVOYER_HOOK_STAGING: ${{ secrets.ENVOYER_HOOK_STAGING }}
        if: github.ref == 'refs/heads/staging'
        run: curl "$ENVOYER_HOOK_STAGING"?sha=${{ github.sha }}

      - name: Deploy to production
        env:
          ENVOYER_HOOK_PRODUCTION: ${{ secrets.ENVOYER_HOOK_PRODUCTION }}
        if: github.ref == 'refs/heads/production'
        run: curl "$ENVOYER_HOOK_PRODUCTION"?sha=${{ github.sha }}

SECRETS

Our .env.github_action file we just use for the Github Actions. Make sure you're creating Github SECRETS for the following values:

[ ] AWS_ACCESS_KEY_ID
[ ] AWS_BUCKET
[ ] AWS_DEFAULT_REGION
[ ] AWS_SECRET_ACCESS_KEY
[ ] ENVOYER_HOOK_DEVELOPER
[ ] ENVOYER_HOOK_PRODUCTION
[ ] ENVOYER_HOOK_STAGING

.env.github_action

APP_NAME="Lasso Project"
APP_ENV=testing
APP_KEY=
APP_DEBUG=false
APP_URL=http://localhost:8000

LOG_CHANNEL=stack
DB_CONNECTION=sqlite
FILESYSTEM_DRIVER=public

BROADCAST_DRIVER=log
CACHE_DRIVER=file
QUEUE_CONNECTION=sync
SESSION_DRIVER=database
SESSION_LIFETIME=120

AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_DEFAULT_REGION=eu-central-1
AWS_BUCKET=content-codebar
AWS_PATH=project_path

Laravel Envoyer

After the install composer dependencies we've added the following steps.

cd {{release}} 
php artisan lasso:pull

cd {{release}} 

php artisan migrate --force
php artisan view:cache
php artisan route:cache
php artisan config:cache
php artisan queue:restart

I guess thats +/- for the moment. I'll maybe create a blog post about it as soon We've some time to polish the Github Actions Workflow .yml.

Again many thanks for providing this package! I'm happy to report my experiences as soon as I've tested it a few more weeks.

Cheers Sebastian

Sammyjo20 / lasso