Keycap set upload hook with GeekHack Spy

[summerqm]

I saw some discussions in https://github.com/noudadrichem/keycapsets.com/issues/18 and had a quick chat with @noudadrichem on discord -- figured I would document some of my thoughts below:

Background:

• After stalking r/mk for some times, I noticed a lot of people (new comers and enthusiasts a like) still have trouble discovering new sets/keep up with the current status of a given keycap set.
• After stalking Keycap Designer discord, I noticed one of the most frequently asked questions is "[picture] is this set similar to any other sets?"
• After stalking GeekHack for a while, I noticed there are about a half dozen new ICs for keycap designs showing up every week, but it is a lot of work to filter through the forum to get what you need/stay on top of things.
• Keycapsets.com was created to help designers and consumers to discover ideas and existing sets, but a not up-to-date website (while still very very useful) is not going to solve the above problems.
• Crowd sourcing for sets upload was a good start but without moderators, this could lead to bad actors uploading undesirable contents.

Problem Statement:

• To implement a way to automatically/semi-automatically fetch new keycap sets info from reliable sources, and allow admins/moderators to edit a few details before publishing the new set.

Proposed Solution:

• Based on my observation, GeekHack is one of the most popular site for people to post their interest checks and group buy info.
• GeekHack Spy shows the latest posts/replies on the forum (and it looks like it can be filtered down to specific boards, e.g. https://geekhack.org/index.php?action=recenttopics&board=132.0)
• There are a few data points that can be fetched from GH Spy: topic, creator, created date, post url -- these data could be used for the initial entry point to the database (with post url as the key). IC/GB topic format are pretty standard, so it can be parsed out using some regex to retrieve the status (IC/GB), profile (GMK/ePBT/KAT/SA/etc.), and name of the set. The regex can also help to filter out not keycap related topics like IC/GB for keyboard/deskmat/switches. In case there are topics that did not follow the standard format and fall through, it can be manually added via some other means (not part of the scope of this issue).
• Once the initial record is created, the status of the record will be set to pending mod review, which allows mods to filter the database by this status to figure out what sets need to be looked at. Mods can then fill in more details for the set, such as (this part is based on the screenshot provided by noudadrichem): status (IC/GB/Post GB, can be pre-populated from GH Spy input), name (GH topic), website URL (GH url), designer name (GH creator), GB start/end date, cover image URL, description, vendors, material, brand, profile (parsed from GH topic), additional image urls for renders and kits.
• After the mods have filled in the details, the mods can publish the set by changing the status to published (btw all status names are just suggestions), in which the set is now viewable on the website to public.
• Whenever there's a new update/reply from GH, and the topic has changed, then the status of the corresponding set could be updated to be published with updates. Mods can filter by this status to cross check the current set in the database vs. the GH update, and adjust the info (adding vendors, kits, renders, GB date, etc.) as needed. Once the info is updated, the status can be changed back to published.

Nice to Have:

• When a GB post is made, it would be nice to search the existing data for similar name. If a similar name exists and the status of the record is in IC, then instead of creating a new record, it could update the status of the record to IC updated with GB which mods can filter and update the status info accordingly.

[noudadrichem]

Haven't replied to this but definitely working on this stuff! All great idea, thank you! I'm soon opening up the upload form with the 'pending mod' status. "draft" is currently a set status already. Automating some uploads with 'pending mod' is a better way to start! Thank you.

But: Scraping Geekhack is a hard thing to do as KCS is unique in its way that is populates a lot of data for a single keycap set. A lot more then MGB and Keycaplendar are doing. (both great projects!).

The nice to have is also an issue. What defines the 'status' of a GB? Is it only IC or GB or should this also take into account some of the 'rounds' a Keycapset has as some designs changes per round they run.

Yet again, thanks for thinking with me! I'm doing my best to create a platform that helps most of the people.