Closed KTibow closed 3 years ago
bessarabov
commented
4 years ago Oh! I really like this idea, thank you! (I know that there is an api that you can use to search the whole github, but I have never used it and I didn't thought about using it for cards search)
I think that as the first step I will find all repos that just have hacs.json file in the root and manually check all of them that are not listed in this repo.
Thank you! It is a really great idea I had not thought about.
KTibow
commented
4 years ago I'm working on something right now.
KTibow
commented
4 years ago Also you should continusly check the defaults list, and the custom-cards org.
KTibow
commented
4 years ago Whipped something up in Python. Requires requests. Change the range(10) to how many pages you want to scan. Also change the PAT.
print("Loading...")
import requests
import json
from time import sleep
for page in range(10):
resp = requests.get(
"https://api.github.com/search/code",
headers={
"Authorization": "token REMOVED!!!!!"
},
params={"q": "filename:hacs.json", "page": str(page)},
)
sleep(2)
resp = resp.json()["items"]
for result in resp:
reponame = result["repository"]["full_name"]
if (
"component" not in reponame
and "home-assistant-community-themes" not in reponame
):
print(".", end="")
repopart = result["repository"]["name"]
query = "repo:" + reponame
query += " extension:js"
query += " filename:" + repopart + ".js"
if "lovelace-" in repopart and repopart[:9] == "lovelace-":
query += (
" filename:" + repopart.replace("lovelace-", "") + ".js"
)
files = requests.get(
"https://api.github.com/search/code",
headers={
"Authorization": "token REMOVED!!!!!"
},
params={"q": query},
)
files = files.json()
if files["total_count"] > 0:
print("\n" + reponame)
sleep(2)Still WIP
KTibow
commented
4 years ago Improved a lot more:
print("Searching...")
import requests
import json
from datetime import datetime
fromisoformat = datetime.fromisoformat
now = datetime.now
from time import sleep
donestuff = 0
for page in range(10):
resp = requests.get(
"https://api.github.com/search/code",
headers={
"Authorization": "token whyyawannaknow"
},
params={"q": "filename:hacs.json", "page": str(page)},
)
sleep(1)
resp = resp.json()["items"]
for result in resp:
reponame = result["repository"]["full_name"]
if (
"component" not in reponame
and "home-assistant-community-themes" not in reponame
):
repopart = result["repository"]["name"]
commits = requests.get(
"https://api.github.com/repos/" + reponame + "/commits",
headers={
"Authorization": "token whyyouwannaknow"
},
)
commits = commits.json()
commitsha = commits[0]["sha"]
files = requests.get(
"https://api.github.com/repos/"
+ reponame
+ "/git/trees/"
+ commitsha,
headers={
"Authorization": "token whyyouwannaknow"
},
params={"recursive": "1"},
)
files = files.json()
files = [file["path"] for file in files["tree"]]
files = [
file.split("/")[len(file.split("/")) - 1] for file in files
]
works = False
for file in files:
if file.lower() == repopart.lower() + ".js":
works = True
if "lovelace-" in repopart and repopart[:9] == "lovelace-":
if (
file.lower()
== repopart.replace("lovelace-", "").lower() + ".js"
):
works = True
if works:
info = requests.get(
"https://api.github.com/repos/" + reponame,
headers={
"Authorization": "token whyyouwannaknow"
},
)
info = info.json()
last_updated = fromisoformat(info["pushed_at"][:-1])
last_updated = last_updated.timestamp()
if now().timestamp() - last_updated < 60 * 60 * 24 * 30 * 6:
print(reponame)
with open("cards.txt", "a") as cards:
cards.write(reponame + "\n")
else:
print("Excluding", reponame)
sleep(1)After searching the interwebz here's what I found after removing duplicates:
ofekashery/vertical-stack-in-card
thomasloven/lovelace-dummy-entity-row
dnguyen800/air-visual-card
DBuit/cover-popup-card
r-renato/ha-card-weather-conditions
pasleto/lovelace-roomba-e5-card
shaonianzhentan/lovelace-ha-iframe-panel
Villhellm/lovelace-animated-background
ljmerza/reddit-card
thomasloven/card-tools
MisakaSpace/tv-card-phoenix
piitaya/lovelace-climate-mode-entity-row
bessarabov/animated-consumption-card
thomasloven/lovelace-slider-entity-row
r-renato/ha-card-waze-travel-time
dintskirveli/lovelace-mta-subway-realtime-card
JH-Soft-Technology/acond-heat-card
hasl-sensor/lovelace-hasl-traffic-status-card
fineemb/lovelace-dc1-card
gurbyz/power-wheel-card
faeibson/lovelace-multiline-text-input-card
amaximus/garbage-collection-card
AmoebeLabs/flex-horseshoe-card
tomvanswam/compass-card
thomasloven/lovelace-q-card
GrecHouse/korean-workday-card
finity69x2/cover-control-button-row
finity69x2/cover-position-preset-row
burnnat/lovelace-template-glance-card
summerscar/lovelace-lights-card
DBuit/image-viewer-card
Yannik25/bayernluefter-lovelace-card
konnectedvn/lovelace-vertical-slider-cover-card
DBuit/screensaver-card
Deejayfool/hass-shutter-card
Villhellm/custom-sidebar
benct/lovelace-github-entity-row
rsnodgrass/water-heater-card
au190/au190_bkk_stop_card
marrobHD/tv-card
ljmerza/calendar-card
Sian-Lee-SA/honeycomb-menu
Mariusthvdb/custom-ui
tommyjlong/wral-weather-card
juagarh5/blind-control-card
au190/au190_thermostat_card
lorenzlars/lovelace-thermostat-preset-row
DBuit/light-popup-card
finity69x2/light-brightness-preset-row
custom-cards/spotify-card
gadgetchnnel/lovelace-card-templater
ben8p/lovelace-tab-redirect-card
nielsfaber/scheduler-card
Villhellm/lovelace-clock-card
hasl-sensor/lovelace-hasl-departure-card
BlueBlueBlob/lovelace-gtasks-card
toringer/lovelace-yr-card
cbulock/lovelace-battery-entity
thomasloven/lovelace-template-entity-row
fineemb/lovelace-air-filter-card
finity69x2/binary-control-button-row
Anonym-tsk/lovelace-starline-card
DBuit/sonos-card
marrobHD/firetv-card
badguy99/PlantPictureCard
dnguyen800/spotify-playlist-card
Imbuzi/meteo-france-weather-card
dimagoltsman/generic-remote-control-card
iwzoo/lovelace-xiaomi-purifier
au190/au190_lock_entity
chaptergy/lightalarm-card
thomasloven/lovelace-badge-card
abmantis/ozw-network-visualization-card
thomasloven/lovelace-state-switch
ljmerza/light-entity-card
toringer/lovelace-nrk-news-card
shaonianzhentan/ha_sidebar
benct/lovelace-xiaomi-vacuum-card
danimart1991/pvpc-hourly-pricing-card
slimcdk/lovelace-button-entity-row
marrobHD/rotel-card
custom-cards/button-entity-row
madmicio/lights-panel-card
gadgetchnnel/lovelace-card-preloader
Cultti/timer-card
reptilex/tesla-style-solar-power-card
vanstinator/sprinkler-card
thomasloven/lovelace-fold-entity-row
tcarlsen/lovelace-light-with-profiles
iwzoo/lovelace-xiaomi-smartfan
RodBr/miflora-card
fineemb/lovelace-fan-xiaomi
DBuit/thermostat-popup-card
KTibow/fullscreen-card
pasleto/lovelace-iphone-xs-card
marksie1988/atomic-calendar-revive
sahulkrishan/toggle-card
xMrVizzy/button-toolbar
cubukun/lich-dam-gio
Savjee/button-text-card
thomasloven/lovelace-gap-card
peetereczek/ztm-stop-card
bramkragten/swipe-card
custom-cards/flex-table-card
dhanani94/mbta-card
TheLastProject/lovelace-valetudo-map-card
thomasloven/lovelace-layout-card
thomasloven/lovelace-card-mod
jeremywillans/lovelace-blank-card
jouke-dev/clock-card
custom-cards/surveillance-card
moralmunky/Home-Assistant-Mail-And-Packages-Custom-Card
hulkhaugen/hass-bha-icons
tommyjlong/doorvivint-card
thomasloven/lovelace-card-tools
pfunkmallone/HACS-camect-custom_card
thomasloven/lovelace-hui-element
denysdovhan/vacuum-card
DBuit/switch-popup-card
royto/linky-card
DBuit/Homekit-panel-card
DBuit/blinds-tile-card
DBuit/media_player-popup-card
dimagoltsman/refreshable-picture-card
custom-cards/dual-gauge-card
madmicio/shutter-cover-panel-card
Townhaus/spu-trash-collection-card
lukevink/lovelace-buien-rain-card
jeremywillans/lovelace-roomba-vacuum-card
fineemb/lovelace-cn-map-card
Imbuzi/lovelace-kiosk
Voxxie/lovelace-jumbo-card
custom-cards/rmv-card
madmicio/ampli-panel-card
mattieha/select-list-card
andrew-codes/shopping-list-card
ljmerza/tracking-number-card
fineemb/lovelace-remote-card
michaelblight/lovelace-digital-clock
miumida/lotto645-card
thomasloven/lovelace-more-info-card
ljmerza/travel-time-card
custom-cards/favicon-counter
azuwis/zigbee2mqtt-networkmap
ricreis394/chartjs-card
benct/lovelace-multiple-entity-rowThis didn't scan repos that don't contain hacs.json files. Also there's more repos with hacs.json files then you think, there's actually 1,101 and counting.
Anyway here's the code:
print("Searching...")
import requests
import json
from datetime import datetime
fromisoformat = datetime.fromisoformat
now = datetime.now
from time import sleep
donestuff = 0
for page in range(150):
resp = requests.get(
"https://api.github.com/search/code",
headers={
"Authorization": "token imlazy"
},
params={"q": "filename:hacs.json", "page": str(page)},
)
sleep(1)
resp = resp.json()
if "items" in resp:
resp = resp["items"]
else:
sleep(60)
print("Page:", page)
print(resp)
exit()
for result in resp:
reponame = result["repository"]["full_name"]
if (
"component" not in reponame
and "home-assistant-community-themes" not in reponame
and "integration" not in reponame
):
repopart = result["repository"]["name"]
commits = requests.get(
"https://api.github.com/repos/" + reponame + "/commits",
headers={
"Authorization": "token imlazy"
},
)
commits = commits.json()
commitsha = commits[0]["sha"]
files = requests.get(
"https://api.github.com/repos/"
+ reponame
+ "/git/trees/"
+ commitsha,
headers={
"Authorization": "token imlazy"
},
params={"recursive": "1"},
)
files = files.json()
files = [file["path"] for file in files["tree"]]
files = [
file.split("/")[len(file.split("/")) - 1] for file in files
]
works = False
for file in files:
if file.lower() == repopart.lower() + ".js":
works = True
if "lovelace-" in repopart and repopart[:9] == "lovelace-":
if (
file.lower()
== repopart.replace("lovelace-", "").lower() + ".js"
):
works = True
if works:
info = requests.get(
"https://api.github.com/repos/" + reponame,
headers={
"Authorization": "token imlazy"
},
)
info = info.json()
last_updated = fromisoformat(info["pushed_at"][:-1])
last_updated = last_updated.timestamp()
if (
now().timestamp() - last_updated < 60 * 60 * 24 * 365
and not info["archived"]
):
print(reponame)
with open("cards.txt", "a") as cards:
cards.write(reponame + "\n")
else:
print(reponame, "EXCLUDED")
sleep(3.3)With this code:
After 1.32902417 hours of scouring 1,000 repositories, I found >100 repos. Please manually look at each one.
cards.txt
I couldn't get the oldest 101 cards, but here are some that either haven't been commited to a while or are archived:
oldcards.txt
bessarabov
commented
4 years ago Thank you!
I've taken the files cards.txt, oldcards.txt and parsed it. 77 repos from that files are already in this repo, but 73 are not. I have checked all that 73 repos manually and I have added 29 repos to this repo — https://github.com/bessarabov/home-assistant-cards/commit/0c270df5719cf9aadbab426935244fcd9e34b9d9 (this is already visible at https://home-assistant-cards.bessarabov.com/ )
There are several reasons why I don't add that repos:
I'm pretty sure that I have not added some cards that I should have added.
Thank you for the idea and for the implementation!
bessarabov
commented
3 years ago I have implemented the system that can be called "automated scanning".
I have written a script that is searching for github repositories that have words in readme: "home assistant custom card".
Here is the sample request to API:
curl "https://api.github.com/search/repositories?q=home%20assistant%20custom%20lovelace%20card%20in:readme%20archived:false&sort=stars&page=1&per_page=3" \
-H 'Accept: application/vnd.github.v3+json'The script is showing all the repos that it has found except the repos that are already in https://github.com/bessarabov/home-assistant-cards and that are not skipped (there is a text file with the list of skipped repos)
I've run this script and I've got about 600 repos. I have put all of them in the skipped file. When the new repo appear the new run of this script will show the new repos and I will add it to this repo if it is actually a card, or I will add it to the skip file.
I will manually run this script from time to time and it will help finding newly created cards.
I will also check all the repos that were found on script first run and add all the cards, but it will take a lot of time.
The source code of the script — https://gist.github.com/bessarabov/b64c8de53a005656aef7626f7a28e346
This is not 100% automation, but it is good enough for the current situation.
Here's an idea: Write a program that queries the GitHub API for all
hacs.jsonfiles, verifies they're valid (right place, right stuff, is a card and not anything else), and then makes sure the last commit was more recent than 6 months ago.