bpepple
commented
8 months ago Been working on a solutions to allow more users to add info thru the Post API. There are a few problems with opening up to all users:
- Quality of data I had a few users send me their metadata from a large portion of their collections, and based on that sampling, I was convinced it's not a great solution.
- Prevention of duplicate data Another problem is the user would need to query the Metron API to determine whether new data (creators, characters, etc) needs to be added. Don't think I want each user writing the code to handle this checking, so some common tool would be needed and usage enforced.
- Increased Server Load Allowing everyone to use the Post API would greatly increase the load on the server since just creating an issue would need a minimum of 2 Post request (1 creating the issue, 1 adding the creator credits), and that's not including any API calls to verify data. Theoretically, a user could add around 900 issues max an hour. When you consider how many users could do that in addition to the current usage, I would most likely need to seriously reduce the current API rate, or spend significantly more upgrading the servers.
The current solution I'm working towards is allowing a few trusted contributors to use the internal tool I've written (Barda) to add new issues. This solves the above problems, since Barda gathers information from a variety of source (Marvel, GCD, etc), uses a key database to link sources (character, creators, etc) to prevent duplication from being created, and then cleans the data before posting to Metron. By having a small subset of users, I also can control the server QOS.
Anyway, if you've got any additional questions, don't hesitate to ask.
Oh, and the git repo for the website is located here.
Is your feature request related to a problem? Please describe. Not related to a bug but would allow for creation of automated pulling from contributor's backlog series/issues with metadata from various sources which may in turn increase contributions to the dataset.
Describe the solution you'd like An API for PUT calls of all metadata related fields including ability to create series and feed new series ID to allow filling of all subsidiary info into the newly created series.
Suggested reference implementation: Tagged library is parsed by shell/python script for metadata and series name API call to cross check series' existence, auto-create with series name sent on API call and come back with new/existing series ID as applicable Update each issue with 1 set of any metadata existing in the comic archives - so 1 API call for all metadata items of each issue (optional but should be heavily encouraged) 1 more API call per issue to send the 1st page of the archive/Cover. This should have a 90% success rate or more with the people who are likely to share their metadata being usually serious collectors. Move to next series once complete.
I do recommend each contributed series to go into the editorial queue, similar as they may for bulk website changes.
Describe alternatives you've considered Anything but entering in (in my case) 41k issues by hand and uploading covers by hand.
Additional context Many apologies if this is the wrong github issues page, I did look for the main project's page but this is the closest project to relevant I could find.