Update GitHub Action to support new crawler v3

[kiliman]

Make sure the action uses the latest version of the crawler, as v2 will not properly parse the updated Tailwind UI site.

/cc @GregBrimble

[ademers]

I added an .env file with:

EMAIL=******
PASSWORD=******
OUTPUT=$HOME/Projects/oss/tailwindui
LANGUAGES=html,react,vue  # default is html
BUILDINDEX=1    # 0 | 1

I ran the action from GH. However I only see the html & react components, and not the vue ones. Am I missing something? Thanks!

[kiliman]

That is odd. I just re-ran it (from a clean output folder) and get all the vue components.

NOTE: I did this from a local install. What happens if you run the crawler locally?

[ademers]

Was using my existing private repo. Maybe it's best I start over from scratch.

[heychazza]

I was also using my existing repo and got same results, will give it a shot with a clean repo later..

[GregBrimble]

I'm currently just pulling the latest changes and figuring out what (if any) changes need to be made :) Gimme about 30 mins – 1 hour, and I should have it all working and updated.

[GregBrimble]

New v1.1.0 release has been published: https://github.com/marketplace/actions/crawl-tailwind-ui

Just update your workflow files from

# ...
      - name: Run crawler
-        uses: gregbrimble/tailwindui-crawler-action@v1.0.0
+        uses: gregbrimble/tailwindui-crawler-action@v1.1.0
        with:
          email: ${{ secrets.TAILWINDUI_EMAIL }}
          password: ${{ secrets.TAILWINDUI_PASSWORD }}

That should be all that's required for old repositories with a master default branch. If you're using a new repository, or have changed your default branch, you'll need to specify branch like so:

# ...
      - name: Run crawler
        uses: gregbrimble/tailwindui-crawler-action@v1.1.0
        with:
          email: ${{ secrets.TAILWINDUI_EMAIL }}
          password: ${{ secrets.TAILWINDUI_PASSWORD }}
+         branch: main

I'll update this project's README.md in just a couple of minutes.

[kiliman]

Hmm... first time trying the Action and I'm seeing the following:

If I choose 'alpine' for languages, I'm saving the preview HTML. However, I'm getting different results when running locally vs running in the action. It's auto-closing tags and stripping HTML entities. Red is local version and green is from action.

I'm not sure why it would return different HTML. Unfortunately the action version is incorrect as the HTML entities are critical. Especially with embedded quotes "

[kiliman]

I updated the package versions (pushed as v3.1.3) and it now works properly on the GitHub Action.

Thanks for taking care of this!

[heychazza]

Morning both,

Had an issue originally which was fixed by creating a new repo (very strange). However I have now tried again, and its only creating the html version..

Using the TailwindUI Crawler.

.env

LANGUAGES=html,react,vue,alpine # defaults to html
BUILDINDEX=1  # generate index file to view components offline

.github/workflows/default.yml

name: Update
on:
  workflow_dispatch: # On manual trigger
  schedule:
    - cron: "0 0 * * *" # Every day at midnight

jobs:
  update:
    name: Update
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Run crawler
        uses: gregbrimble/tailwindui-crawler-action@v1.1.0
        with:
          email: ${{ secrets.TAILWINDUI_EMAIL }}
          password: ${{ secrets.TAILWINDUI_PASSWORD }}
          branch: main

Also, weirdly it's using "root" rather than my own avatar.

cc: @kiliman @GregBrimble

Thanks a ton to you both for creating both things, been incredibly useful.

[GregBrimble]

@heychazza , try also adding current_branch: main as well as branch: main.

You're seeing html only because that's the crawler default, and it's not picking up the .env in the current_branch which defaults to master.

Sorry, I should have made that clearer.

[heychazza]

@heychazza , try also adding current_branch: main as well as branch: main.

You're seeing html only because that's the crawler default, and it's not picking up the .env in the current_branch which defaults to master.

Sorry, I should have made that clearer.

You're a star! Thank you so much :)

It runs (I ended up just using a new repo aha), however Alpine version is non existent though the others have (html, vue and react).

Maybe more of a issue with the crawler itself rather than the action? Once again, thanks to you both a ton for building these and being fast with responses 👍

[GregBrimble]

Do you pin the crawler version with crawlerref? I think Alpine support was added as a new release. By default, my Action should be pulling master.

Can you reproduce locally with that exact same .env file?

[heychazza]

Hey buddy,

.env

LANGUAGES=html,react,vue,alpine # defaults to html
BUILDINDEX=1  # generate index file to view components offline

workflow:

name: Update
on:
  workflow_dispatch: # On manual trigger
  schedule:
    - cron: "0 0 * * *" # Every day at midnight

jobs:
  update:
    name: Update
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Run crawler
        uses: gregbrimble/tailwindui-crawler-action@v1.1.0
        with:
          email: ${{ secrets.TAILWINDUI_EMAIL }}
          password: ${{ secrets.TAILWINDUI_PASSWORD }}
          branch: main
          current_branch: main
          actor: heychazza

I don't currently use crawlerref

[GregBrimble]

@heychazza , I can't reproduce. Alpine components get pulled as expected for me. Any chance you can share your repo with me so I can take a look at the Action logs, and whatever else might help us find out why this isn't working for you?

[heychazza]

Hey @GregBrimble,

Fixed it by moving alpine.. haha..

From..

LANGUAGES=html,react,vue,alpine # defaults to html
BUILDINDEX=1  # generate index file to view components offline

To..

LANGUAGES=alpine,html,react,vue # defaults to html
BUILDINDEX=1  # generate index file to view components offline

Not sure if Github caches the .env which is the cause here, or the order mattered.

[heychazza]

Ok, so doing that has caused vue to disappear..

📝  Writing html simple_centered.html
📝  Writing react simple_centered.jsx
📝  Writing alpine simple_centered.html

[GregBrimble]

Again, unfortunately can't reproduce. Adding vue and alpine to .env seem to work as expected for me. Can you share your repository with me?

[heychazza]

Again, unfortunately can't reproduce. Adding vue and alpine to .env seem to work as expected for me. Can you share your repository with me?

Do you have both packages with TUI?

[GregBrimble]

Yup.

[heychazza]

Yup.

Provided you access!

[heychazza]

Yup.

https://github.com/track/tailwindui.com/invitations

[GregBrimble]

I have no idea why, but:

- LANGUAGES=alpine,html,react,vue # defaults to html
- BUILDINDEX=1  # generate index file to view components offline
+ LANGUAGES=html,react,alpine,vue
+ BUILDINDEX=1

Fixed it.

That's gotta be something from the crawler itself, rather than my Action.

[GregBrimble]

Yeah, just locally tested an .env which looks like this, and it only pulled Alpine, HTML & React:

EMAIL=my@email.address
PASSWORD=Sup3r5ecr3tP@ssw0rd!
LANGUAGES=alpine,html,react,vue # defaults to html
BUILDINDEX=1  # generate index file to view components offline

Something about the way the crawler parses the .env files, @kiliman ?

[heychazza]

Yeah, just locally tested an .env which looks like this, and it only pulled Alpine, HTML & React:

EMAIL=my@email.address
PASSWORD=Sup3r5ecr3tP@ssw0rd!
LANGUAGES=alpine,html,react,vue # defaults to html
BUILDINDEX=1  # generate index file to view components offline

Something about the way the crawler parses the .env files, @kiliman ?

Yeah, seems to only get the first 3 from what it seems..

Edit: oh its the comment?

[heychazza]

https://github.com/motdotla/dotenv/issues/348#issuecomment-454632909

Seems to be an issue currently.. (inline wise)

[kiliman]

Wow, thanks for troubleshooting that.

Yeah, I noticed that inline comments weren't supported in .env files when I was testing. I just thought it was common knowledge, and that I was making a dumb assumption. The comments in the examples were more for documentation.

I will update the README to move the comments to a separate line, and also will make note of it in the docs.

Thanks again and I'm glad you're enjoying the tool.

BTW: Those React/Vue components are sweet. Tailwind Labs are on 🔥

[heychazza]

Wow, thanks for troubleshooting that.

Yeah, I noticed that inline comments weren't supported in .env files when I was testing. I just thought it was common knowledge, and that I was making a dumb assumption. The comments in the examples were more for documentation.

I will update the README to move the comments to a separate line, and also will make note of it in the docs.

Thanks again and I'm glad you're enjoying the tool.

BTW: Those React/Vue components are sweet. Tailwind Labs are on 🔥

Haha, tbh mate - I fell for it as well and kept the inline comments (I didn't know! aha). I defo think just moving comments is all thats needed, as I just copy pasted it.

I agree, they are doing so fantastic now, pushing lots of great content all round 👍 Off topic, @adamwathan @steveschoger fantastic work guys (and the team!).

[ademers]

I can confirm that removing the inline comments from the .env worked in my existing, and brand new repos.

EMAIL=******
PASSWORD=******
OUTPUT=$HOME/Projects/oss/tailwindui
LANGUAGES=alpine,html,react,vue
BUILDINDEX=1

Outputs

[heychazza]

Hey both, only odd thing I find is..

# OUTPUT optional, defaults to ./output
OUTPUT=/path/to/output

If I don't specify this, it outputs the same as the above? Is this a bug that it isn't outputting into the output folder?

[kiliman]

Hmm.. the code looks like this:

const output = process.env.OUTPUT || './output'

If you don't provide OUTPUT then it should create a folder called output in your current directory, and store everything there: /output/html, /output/react, ...

So what are you seeing?

In my GitHub Action, I set OUTPUT=. because I want it to output into the root of the project.

[heychazza]

Hmm.. the code looks like this:

const output = process.env.OUTPUT || './output'

If you don't provide OUTPUT then it should create a folder called output in your current directory, and store everything there: /output/html, /output/react, ...

So what are you seeing?

In my GitHub Action, I set OUTPUT=. because I want it to output into the root of the project.

So, my .env is:

# LANGUAGES defaults to html
LANGUAGES=html,react,vue,alpine
# BUILDINDEX generate index file to view components offline
BUILDINDEX=1

It shows as separate folders weirdly.

---

A github action bug as well, it's using "root" with no avatar as the author, shouldn't it be my github account?

[kiliman]

Not sure why it's not creating a top-level output folder.

But yes, v3 now creates separate directory trees for each language.

As for the commit user, it's working for me as kiliman. I even modified the commit message.

Look at the documentation for the action. You can specify the actor name (the commit user) although for me it defaulted to my username.

https://github.com/gregbrimble/tailwindui-crawler-action

my default.yml file

name: Update
on:
  workflow_dispatch:
  schedule:
    - cron: '0 0 * * *' # Every day at midnight

jobs:
  update:
    name: Update
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Run crawler
        uses: gregbrimble/tailwindui-crawler-action@v1.1.0
        with:
          email: ${{ secrets.TAILWINDUI_EMAIL }}
          password: ${{ secrets.TAILWINDUI_PASSWORD }}
          commitmessage: 🤖 Update Tailwind UI components from Action

my .env file

OUTPUT=.
LANGUAGES=html,react,vue,alpine
BUILDINDEX=1

v3 output directory structure

.
├── alpine
│   └── components
│       ├── application-ui
│       │   ├── application-shells
│       │   │   ├── multi-column
│       │   │   │   ├── constrained_three_column.html
│       │   │   │   ├── full_width_with_narrow_sidebar.html
│       │   │   │   └── full_width_with_narrow_sidebar_and_secondary_column_on_right.html
├── html
│   └── components
│       ├── application-ui
├── preview
│   ├── components
│   │   ├── application-ui
│   │   │   ├── application-shells
│   │   │   │   ├── multi-column
│   │   │   │   │   └── index.html
│   │   │   │   ├── sidebar
│   │   │   │   │   └── index.html
│   │   │   │   └── stacked
│   │   │   │       └── index.html
├── react
│   └── components
│       ├── application-ui
│       │   ├── application-shells
└── vue
    └── components
        ├── application-ui
        │   ├── application-shells
        │   │   ├── multi-column
        │   │   │   ├── constrained_three_column.vue

[heychazza]

Not sure why it's not creating a top-level output folder.

But yes, v3 now creates separate directory trees for each language.

As for the commit user, it's working for me as kiliman. I even modified the commit message.

Look at the documentation for the action. You can specify the actor name (the commit user) although for me it defaulted to my username.

https://github.com/gregbrimble/tailwindui-crawler-action

my default.yml file

name: Update
on:
  workflow_dispatch:
  schedule:
    - cron: '0 0 * * *' # Every day at midnight

jobs:
  update:
    name: Update
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Run crawler
        uses: gregbrimble/tailwindui-crawler-action@v1.1.0
        with:
          email: ${{ secrets.TAILWINDUI_EMAIL }}
          password: ${{ secrets.TAILWINDUI_PASSWORD }}
          commitmessage: 🤖 Update Tailwind UI components from Action

my .env file

OUTPUT=.
LANGUAGES=html,react,vue,alpine
BUILDINDEX=1

v3 output directory structure

.
├── alpine
│   └── components
│       ├── application-ui
│       │   ├── application-shells
│       │   │   ├── multi-column
│       │   │   │   ├── constrained_three_column.html
│       │   │   │   ├── full_width_with_narrow_sidebar.html
│       │   │   │   └── full_width_with_narrow_sidebar_and_secondary_column_on_right.html
├── html
│   └── components
│       ├── application-ui
├── preview
│   ├── components
│   │   ├── application-ui
│   │   │   ├── application-shells
│   │   │   │   ├── multi-column
│   │   │   │   │   └── index.html
│   │   │   │   ├── sidebar
│   │   │   │   │   └── index.html
│   │   │   │   └── stacked
│   │   │   │       └── index.html
├── react
│   └── components
│       ├── application-ui
│       │   ├── application-shells
└── vue
    └── components
        ├── application-ui
        │   ├── application-shells
        │   │   ├── multi-column
        │   │   │   ├── constrained_three_column.vue

Thanks a ton, will give it a shot now! Any reason why I'm showing as "root" ? I even specified the author to use with the action,

Is there any way I can use v3 or is that the current default?

Will give your .env a shot.

[heychazza]

Weird, using your exact .env and workflow file but added OUTPUT=./output and it's not respecting the path..

[kiliman]

Yeah, everyone must upgrade to the new v3 crawler. The Tailwind UI web site has changed its layout, so the older v2 crawler no longer parses the HTML properly.

This is a risk when "screen-scraping"... always chasing the dragon 🐉

[kiliman]

Check your action log. The crawler will display the current OUTPUT folder. Remember, it's relative to the project root, so in this case, I'm running from my tailwind-crawler project and want to output to a sibling project folder tailwindui

OUTPUT=../tailwindui
LANGUAGES=html,react,vue,alpine
BUILDINDEX=1

[heychazza]

Check your action log. The crawler will display the current OUTPUT folder. Remember, it's relative to the project root, so in this case, I'm running from my tailwind-crawler project and want to output to a sibling project folder tailwindui

OUTPUT=../tailwindui
LANGUAGES=html,react,vue,alpine
BUILDINDEX=1

Could the author thing be due to it being from an organisation? Even though I did specify actor in the action workflow file..

[kiliman]

Hmm... I wonder it there's an issue because it's using /output (absolute path) vs ./output (relative path, notice period)

Although that's what the default should be ./output, what happens if you explicitly specify OUTPUT=./output (make sure to include the leading period)?

[heychazza]

Hmm... I wonder it there's an issue because it's using /output (absolute path) vs ./output (relative path, notice period)

Although that's what the default should be ./output, what happens if you explicitly specify OUTPUT=./output (make sure to include the leading period)?

Oddly.. I am..

OUTPUT=./output
LANGUAGES=html,react,vue,alpine
BUILDINDEX=1

[kiliman]

Interesting, I checked my action log and it's also hard coding /output despite my OUTPUT=., so this may be the action overriding that setting? @GregBrimble

[heychazza]

Interesting, I checked my action log and it's also hard coding /output despite my OUTPUT=., so this may be the action overriding that setting? @GregBrimble

I think the action is overriding most options, as for example actor still shows me as root rather than my username. Perhaps custom options are not being considered?

[kiliman]

Yes, the action is overriding it. Most likely to simplify his scripts for committing, etc. Not sure about the username though.

[heychazza]

Yes, the action is overriding it. Most likely to simplify his scripts for committing, etc. Not sure about the username though.

The odd thing is, even so - it's not applying to that output folder, unless the ./output is needed (like you were mentioning earlier).

[kiliman]

No, I believe he treats the /output folder as the root of your repo. It's where he clones the current repo, crawler updates any files, then he commits any changes.

[heychazza]

No, I believe he treats the /output folder as the root of your repo. It's where he clones the current repo, crawler updates any files, then he commits any changes.

Ahh that makes sense!

[GregBrimble]

Sorry, just catching up on that conversation now:

• Unless you specify a crawlerref input in my GitHub Action, we'll use whatever is on master of @kiliman/tailwindui-crawler by default. So you don't need to manually update to v3—that's done automatically.
• If you remove the actor option, if defaults to {{ github.actor }} which should be you. You can manually override this by specifying actor, but you'll need a PAT which authenticates as that user.
• Yes, the Action defaults to the root of the output repository. It uses /output in the Action workspace, but that's just where we clone the output repository to, so it's not actually reflected in the output repository itself. If you want the option to customize an output directory in your repository, file an issue on the Action, and we can discuss requirements there :) edit: scratch that, already got an issue for that—comment here

[heychazza]

• If you remove the actor option, if defaults to {{ github.actor }} which should be you. You can manually override this by specifying actor, but you'll need a PAT which authenticates as that user.

I tried this, weirdly failed..

Pushing changes...
fatal: unable to access 'https://***/': Could not resolve host: ***

[kiliman]

It might make more sense to add issues to the Action repo, so Greg can track it easier.