Bot Opportunies

[xayhewalo]

Preface

The old Writing Bots Wiki used to have a list of "Bot Opportunities" that have been migrated here. The list migrated from the wiki will likely need further breakdown.

Contributors can edit this issue to track all bot-related clean up

Legend

• A: Authors
• E: Editions
• W: Works

Bot Clean up Opportunities

• [ ] Unicode [A, E, W]
  • 'mojibake' A~(c) style encoding errors. From multiple sources
    • Some from Amazon bulk imports
    • Some from external MARC records Some are lossy, others can be reconstructed. Important point to note: there are multiple styles that look similar, but require different translation to fix.
  • Combining diacritics. Some external MARC records have had subsequent correction, reimportation could catch many such cases internetarchive/openlibrary#150
  • NFC Normalisation. Of titles, description, place, i.e. all text fields. Ensure all input mechanisms normalise correctly. UI + import paths. TEST.
• [X] Spaces in OCAIDS [E] internetarchive/openlibrary#467
• [ ] URLs in OCAIDs [E]
  • not many?
• [ ] URLs in Amazon ids [E]
  • Needs checking. Some url ids appear to resolve correctly. Suspicion that some users are trying to insert affiliate links in OL ids. What is our policy on this? Should we be making more effort to ensure only Internet Archive affiliate links are used?
• [ ] Invalid ISBNs [E]
  • _10 and _13 Other ISBN fields; can they be used for anything?
    • [ ] repair if possible
    • [ ] normalise https://github.com/internetarchive/openlibrary-bots/pull/43#
    • [ ] remove if bad
    • [ ] test for reuse on multiple editions
• [ ] Invalid/Incorrect other ids [E, W]
  • [ ] LCCN internetarchive/openlibrary#2851
  • [ ] OCLC
  • [ ] LibraryThing internetarchive/openlibrary#497
  • Sometimes ids are valid but link to a different book. How to resolve?
• [ ] Empty Titles [A, E, W]
  • Repair if possible
  • Delete if junk data
• [ ] Titles varied only by punctuation or whitespace (where author is the same) [E, W]
  • Remove trailing ".", ":", ";"
  • Reduce whitespaces to single sp
  • When one version has a single sp and another has no sp, change latter to single sp (e.g. Hitchhiker's guide --> Hitch hiker's guide)
  • When one version has embedded punctuation and another does not, change latter to match former (e.g. Hitch hikers guide --> Hitch-hiker's guide)
• [ ] Name Order [A]
  • The Open Library convention is natural name order as opposed to "Last, First" Is everyone happy with this standard? How should titles be handled? e.g. Sir , Lord, Lady, Mrs. There seems to be variation in the data for this style of name.
  • Also, how to handle Aliases? I notice the author_role type https://openlibrary.org/type/author_role has an 'as' property. Is this to allow for name variations on a particular book?
• [ ] Bad Alternate names for authors [A] internetarchive/openlibrary#498
• [ ] Orphaned Editions [E] internetarchive/openlibrary#153
• [ ] Works without Editions [W] internetarchive/openlibrary#2727
  • https://github.com/internetarchive/openlibrary/issues/44 Appears to be a merging task now. These editionless works look to be left over from moving editions (translations are a common example) to another work without converting the original work to a redirect. If there was some way to go back to the the original edition move and complete the work redirection? Perhaps the WorkBot log at https://github.com/internetarchive/openlibrary/blob/629f2afe8fbff9e8af50e00a57e756dff45a0d73/openlibrary/catalog/works/live.py#L438 might be helpful??
• [ ] Editions unlinked to Works [E]
  • Appears to occur mainly with apparently editionless works. How common is it? If rare, it should be fairly simple to address manually in librarian mode given a list of such editions.
• [ ] Authors without Work [A]
• [ ] Orphaned editions without Authors [A]
  • Where author is known to IA, from MARC records. Update the records from the original MARC. Many of the missing authors seem to be on books that have editors rather than straight authors. Is this a cause of the problem? How to correctly represent editors on Open Library works?
• [ ] Non-book Items internetarchive/openlibrary#2404

[BrittanyBunk]

@guyjeangilles I haven't created bots before (I'll look into it more), but I wish there's some description of how bots are created (like what databases they use - do they use the bulk data dump or the API? Idk how to access the API - as the website doesn't say).

[BrittanyBunk]

I think this relates to internetarchive/openlibrary#2902, because without data to be able to be easy to work with - it's going to be difficult to parse through to figure out that bots are needed and how to create bots to work with the data. Bots may also be able to work faster with less data too!

[xayhewalo]

@BrittanyBunk The API documentation could be better but you can most of it here: https://openlibrary.org/developers/api

Also the Bot Wiki is a little outdated but it's a good place to start. @hornc is a good resource as well.

[BrittanyBunk]

@guyjeangilles If that Bot Wiki could be added to the API documentation page on the OL, it would be an enormous help. I think that's the missing connection for explaining how to make bots! It's also nice that there's a new 'read' section too.

[BrittanyBunk]

I am going to list my bot opportunity ideas to keep in place here.

• https://github.com/internetarchive/openlibrary-librarians/issues/10
• https://github.com/internetarchive/openlibrary/issues/2387

[LeadSongDog]

Another one: many author records have a trailing space in the name. These frequently are duplicates of records for the same author sans space. They should be corrected and, where appropriate, merged. See for example https://openlibrary.org/works/OL17769107W