search

tgxn / lemmy-explorer

Instance and Community Explorer for Lemmy
https://lemmyverse.net/
105 stars 9 forks source link

Feature: Upload artifacts to Github Pages #110

Closed Raicuparta closed 1 year ago

Raicuparta commented 1 year ago

What do you want to see? In your GitHub action, serve all the json artifacts via GitHub pages instead of just as zip artifacts in the actions.

Additional context Since you understandably don't want people to hotlink the files from your public website, I'm making my own workflow that just downloads your artifacts and reuploads them to GitHub pages, so they can then be accessed freely. It would be even better if this could be done directly in your workflow, so that both the action artifact and GitHub page are updated simultaneously. I can open a PR for this if you're interested.

tgxn commented 1 year ago

damn that's a good idea. do you have an example workflow? i have them building to assets already so I can copy them into pages I guess.

Raicuparta commented 1 year ago

@tgxn I actually ended up just stealing your crawler code and using only the specific part I needed, and now I'm running the whole thing from within a GitHub workflow. I didn't feel like setting up my own server 😅

Here you can see me running the script with node and then publishing the output to GitHub Pages: https://github.com/Raicuparta/lemmy-community-crawler/blob/main/.github/workflows/deploy.yml

Which results in this: lemmy.raicuparta.com/communities.json

tgxn commented 1 year ago

I am gonna give it a go!

image I could just host the whole dang site on GH pages 🤔

Raicuparta commented 1 year ago

Yep, you totally could. It's not the most efficient though. I guess as long as the whole crawler takes less than 6 hours to finish (free workflow limit), it should be ok.

tgxn commented 1 year ago

I don't think I'll move the whole thing, I'll just add data. for the data...

https://data.lemmyverse.net/data/community.full.json

also made a simple browse thing https://data.lemmyverse.net/

Raicuparta commented 1 year ago

Beautiful. I guess you can consider this one resolved then :)

tgxn commented 1 year ago

soon.

image

just building a simple ui .... make it nicer

tgxn commented 1 year ago

this is great :)

https://data.lemmyverse.net/

closing :)

Raicuparta commented 1 year ago

@tgxn btw just thought you should know, the reason I wanted to use this was for a browser extension I'm making: https://github.com/Raicuparta/lemmy-go

I was initially using browser.feddit.de, but that one seemed either very outdated or missing a lot of large communities for some reason. I was about to make my own crawler from scratch, but I'm glad I found yours, saved me some precious time. So thanks!

tgxn commented 8 months ago

@tgxn btw just thought you should know, the reason I wanted to use this was for a browser extension I'm making: https://github.com/Raicuparta/lemmy-go

I was initially using browser.feddit.de, but that one seemed either very outdated or missing a lot of large communities for some reason. I was about to make my own crawler from scratch, but I'm glad I found yours, saved me some precious time. So thanks!

Good to hear, maybe we should compile a list of apps that use our data.