search

IgnisDa / ryot

Roll your own tracker!
https://ryot.io
GNU General Public License v3.0
2.01k stars 55 forks source link

Goodreads import works for just a few books #1107

Closed arjan-s closed 3 days ago

arjan-s commented 5 days ago

I just exported my csv from Goodreads to import it in my ryot instance, with these results:

Total imported: 63
Failed: 103

And to make it even worse, out of those 63, only 18 are shown on the Books page, and the statistics on the Dashboard page show 13 books.

Details 

[
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "Edgedancer (The Stormlight Archive, #2.5)",
        "error": "Could not convert ISBN: 9781250166548 to Google Books ID"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "The Obelisk Gate (The Broken Earth, #2)",
        "error": "ISBN is empty"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "The Fifth Season (The Broken Earth, #1)",
        "error": "ISBN is empty"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "The Dark Forest (Remembrance of Earth’s Past, #2)",
        "error": "ISBN is empty"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "The Ein Particle",
        "error": "Could not convert ISBN: 9789083233475 to Google Books ID"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "Dit wil je écht niet weten",
        "error": "Could not convert ISBN: 9789463812177 to Google Books ID"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "Barrayar (Vorkosigan Saga, #7)",
        "error": "Could not convert ISBN: 9780743468411 to Google Books ID"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "Red Rising (Red Rising Saga, #1)",
        "error": "ISBN is empty"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "The Three-Body Problem (Remembrance of Earth’s Past, #1)",
        "error": "ISBN is empty"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "The City and the Stars",
        "error": "ISBN is empty"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "Shards of Honor (Vorkosigan Saga, #1)",
        "error": "Could not convert ISBN: 9780671720872 to Google Books ID"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "Magician (The Riftwar Saga, #1-2)",
        "error": "Could not convert ISBN: 9780586217832 to Google Books ID"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "Foreverywhere",
        "error": "ISBN is empty"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "A Big Ship at the Edge of the Universe (The Salvagers, #1)",
        "error": "ISBN is empty"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "The Dead Drink First",
        "error": "ISBN is empty"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "C-Shapes",
        "error": "ISBN is empty"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "The Hitchhiker’s Guide to the Galaxy (The Hitchhiker's Guide to the Galaxy, #1)",
        "error": "ISBN is empty"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "The Forever War (The Forever War, #1)",
        "error": "ISBN is empty"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "Cryptonomicon",
        "error": "ISBN is empty"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "2001: A Space Odyssey (Space Odyssey, #1)",
        "error": "ISBN is empty"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "Rendezvous with Rama (Rama, #1)",
        "error": "Could not convert ISBN: 9781857231588 to Google Books ID"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "Childhood’s End",
        "error": "ISBN is empty"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "I, Robot (Robot, #0.1)",
        "error": "Could not convert ISBN: 9780553803709 to Google Books ID"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "Equal Rites (Discworld, #3; Witches, #1)",
        "error": "Could not convert ISBN: 9780060855901 to Google Books ID"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "Infinite (Infinite, #1)",
        "error": "ISBN is empty"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "Children of Dune (Dune, #3)",
        "error": "Could not convert ISBN: 9780593098240 to Google Books ID"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "Even Tree Nymphs Get the Blues (Mystic Bayou, #3)",
        "error": "ISBN is empty"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "The Mystery of Alice",
        "error": "ISBN is empty"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "Steel World (Undying Mercenaries, #1)",
        "error": "ISBN is empty"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "A Feast of Dragons (The Sorcerer's Ring, #3)",
        "error": "ISBN is empty"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "Retribution (Limitless Lands, #3)",
        "error": "ISBN is empty"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "Opposition (Limitless Lands, #4)",
        "error": "ISBN is empty"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "Conquest (Limitless Lands #2)",
        "error": "ISBN is empty"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "The Man Who Knew The Way to the Moon",
        "error": "ISBN is empty"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "A Quest of Heroes (The Sorcerer's Ring, #1)",
        "error": "ISBN is empty"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "Dune (Dune, #1)",
        "error": "ISBN is empty"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "Children of Ruin (Children of Time, #2)",
        "error": "Could not convert ISBN: 9780316452533 to Google Books ID"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "Rising (Stargate Atlantis, #1)",
        "error": "Could not convert ISBN: 9780954734350 to Google Books ID"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "Trial by Fire (Stargate SG-1, #1)",
        "error": "Could not convert ISBN: 9780954734305 to Google Books ID"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "The Commander's Tale (Limitless Lands #1)",
        "error": "ISBN is empty"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "Where the Hell is Tesla?",
        "error": "ISBN is empty"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "Glorious Appearing: The End of Days (Left Behind, #12)",
        "error": "Could not convert ISBN: 9780842332354 to Google Books ID"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "Armageddon: The Cosmic Battle of the Ages (Left Behind, #11)",
        "error": "Could not convert ISBN: 9780842332361 to Google Books ID"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "2010: Odyssey Two",
        "error": "Could not convert ISBN: 9780345413970 to Google Books ID"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "Time's Eye (A Time Odyssey, #1)",
        "error": "Could not convert ISBN: 9780345452474 to Google Books ID"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "Kroniek van Oderan",
        "error": "Could not convert ISBN: 9789402219241 to Google Books ID"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "Old Man's War (Old Man's War, #1)",
        "error": "ISBN is empty"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "Redshirts",
        "error": "Could not convert ISBN: 9780765316998 to Google Books ID"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "Off to Be the Wizard (Magic 2.0, #1)",
        "error": "Could not convert ISBN: 9781612184715 to Google Books ID"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "The Restaurant at the End of the Universe (The Hitchhiker's Guide to the Galaxy, #2)",
        "error": "ISBN is empty"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "The Waste Lands (The Dark Tower, #3)",
        "error": "Could not convert ISBN: 9780670032563 to Google Books ID"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "Wolves of the Calla (The Dark Tower, #5)",
        "error": "Could not convert ISBN: 9781416516934 to Google Books ID"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "The Dark Tower (The Dark Tower, #7)",
        "error": "ISBN is empty"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "The Gunslinger (The Dark Tower, #1)",
        "error": "ISBN is empty"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "The Return of the King (The Lord of the Rings, #3)",
        "error": "ISBN is empty"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "The Hobbit (The Lord of the Rings, #0)",
        "error": "ISBN is empty"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "The Dispatcher (The Dispatcher, #1)",
        "error": "ISBN is empty"
    },
    {
        "lot": "BOOK",
        "step": "INPUT_TRANSFORMATION",
        "identifier": "We Are Legion (We Are Bob) (Bobiverse, #1)",
        "error": "ISBN is empty"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "Children of Time (Children of Time, #1)",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "Het is oorlog maar niemand die het ziet",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "The Spy and the Traitor: The Greatest Espionage Story of the Cold War",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "The Hacker and the State: Cyber Attacks and the New Normal of Geopolitics",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "Foundation (Foundation, #1)",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "A Memory Called Empire (Teixcalaan, #1)",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "Gateway (Heechee Saga, #1)",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "Pushing Ice",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "The Light Fantastic (Discworld, #2; Rincewind, #2)",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "The Color of Magic (Discworld, #1; Rincewind, #1)",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "A March of Kings (The Sorcerer's Ring, #2)",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "Dune Messiah (Dune, #2)",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "Light of the Jedi (Star Wars: The High Republic)",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "The Silmarillion",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "Under the Dome",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "Ice Station (Shane Schofield, #1)",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "The Remnant (Left Behind, #10)",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "The Mark (Left Behind, #8)",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "The Indwelling (Left Behind, #7)",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "Assassins (Left Behind, #6)",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "Apollyon (Left Behind, #5)",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "Soul Harvest: The World Takes Sides (Left Behind, #4)",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "Tribulation Force (Left Behind, #2)",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "Left Behind (Left Behind, #1)",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "Pandora's Star (Commonwealth Saga, #1)",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "Judas Unchained (Commonwealth Saga, #2)",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "Rama Revealed (Rama, #4)",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "The Garden of Rama (Rama, #3)",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "Rama II (Rama #2)",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "Zero Day (Jeff Aiken, #1)",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "The Secret Sentry: The Untold History of the National Security Agency",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "Sunstorm (A Time Odyssey, #2)",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "Firstborn (A Time Odyssey, #3)",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "The Fountains of Paradise",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "Life, the Universe and Everything (The Hitchhiker's Guide to the Galaxy, #3)",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "Hyperion (Hyperion Cantos, #1)",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "Leviathan Wakes (The Expanse, #1)",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "The Drawing of the Three (The Dark Tower, #2)",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "Wizard and Glass (The Dark Tower, #4)",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "Song of Susannah (The Dark Tower, #6)",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "The Two Towers (The Lord of the Rings, #2)",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "The Fellowship of the Ring (The Lord of the Rings, #1)",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "The Neutronium Alchemist (Night's Dawn, #2)",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "The Naked God (Night's Dawn, #3)",
        "error": "error decoding response body"
    },
    {
        "lot": "BOOK",
        "step": "MEDIA_DETAILS_FROM_PROVIDER",
        "identifier": "The Reality Dysfunction (Night's Dawn, #1)",
        "error": "error decoding response body"
    }
]

I'm not quite sure what to do to help. I already found https://github.com/IgnisDa/ryot/issues/985 which has partly the same problem of the export not containing ISBN numbers for quite a few books. But perhaps the "error decoding response body" could mean some kind of rate limit somewhere when downloading metadata?

Here's my export for completeness: goodreads_library_export.csv

IgnisDa commented 5 days ago

Yes google Books is heavily rate limited. Please use https://docs.ryot.io/guides/books to setup your own api keys.

arjan-s commented 5 days ago

Thanks, that at least answers one of my questions

IgnisDa commented 5 days ago

For books that don't have an ISBN, i am not sure what you want me to do?

arjan-s commented 5 days ago

Apologies, I could indeed have been more specific. It's just that I really like this project so I got too excited trying to get everything to work. :)

I understand the rate limit solution and will look into it. I'm hoping to get an idea of the differences in book amounts, that really strikes me as odd.

Regarding the ISBN numbers, perhaps it's possible to import metadata directly from goodreads? I believe that's what MediaTracker does? Perhaps that could also solve the rate limit issue?

IgnisDa commented 5 days ago

Goodreads doesn't have an API. As for media tracker, it created custom entries for unrecognised books which are not connected to any upstream provider. This means that they don't get updates etc.

i don't like this approach so i decided to just let those items fail. You have 2 options: