search

janeczku / calibre-web

:books: Web app for browsing, reading and downloading eBooks stored in a Calibre database
GNU General Public License v3.0
13.22k stars 1.41k forks source link

[Kobo Sync] Failed to receive or parse response from Kobo's init endpoint. Falling back to un-proxied mode #2901

Closed jfmlima closed 7 months ago

jfmlima commented 1 year ago

Describe the bug/problem

When trying to sync kobo with calibre web I'm getting the error in the title and a 403 on the gettests endpoint nginx | some-ip - - [29/Sep/2023:20:35:23 +0000] "POST /kobo/kobo/TOKEN/v1/analytics/gettests HTTP/1.1" 403 1778

I'm running calibre over nginx on docker hosted in a google compute instance. Everything was running ok until today when I updated my kobo and suddenly I couldn't sync anymore.

I've even tried to remove nginx out of the equation but changing the api_endpoint to go straight to calibre web port to no avail, so I don't think the issue is with any reverse proxy configuration. Accessing to v1/library/sync outputs correctly in the browser.

Logfile [2023-09-29 20:30:58,900] ERROR {cps.kobo:1076} Failed to receive or parse response from Kobo's init endpoint. Falling back to un-proxied mode. [2023-09-29 20:31:50,039] ERROR {cps.kobo:1076} Failed to receive or parse response from Kobo's init endpoint. Falling back to un-proxied mode. [2023-09-29 20:35:21,248] ERROR {cps.kobo:1076} Failed to receive or parse response from Kobo's init endpoint. Falling back to un-proxied mode.

Expected behavior Kobo should sync all everything from calibre as usual.

Environment:

Additional context My database is in google drive, but that shouldn't have any impact on this issue.

This is the output of the /initialization:


  "Resources": {
    "account_page": "https://secure.kobobooks.com/profile",
    "account_page_rakuten": "https://my.rakuten.co.jp/",
    "add_entitlement": "https://storeapi.kobo.com/v1/library/{RevisionIds}",
    "affiliaterequest": "https://storeapi.kobo.com/v1/affiliate",
    "audiobook_subscription_orange_deal_inclusion_url": "https://authorize.kobo.com/inclusion",
    "authorproduct_recommendations": "https://storeapi.kobo.com/v1/products/books/authors/recommendations",
    "autocomplete": "https://storeapi.kobo.com/v1/products/autocomplete",
    "blackstone_header": {
      "key": "x-amz-request-payer",
      "value": "requester"
    },
    "book": "https://storeapi.kobo.com/v1/products/books/{ProductId}",
    "book_detail_page": "https://store.kobobooks.com/{culture}/ebook/{slug}",
    "book_detail_page_rakuten": "https://books.rakuten.co.jp/rk/{crossrevisionid}",
    "book_landing_page": "https://store.kobobooks.com/ebooks",
    "book_subscription": "https://storeapi.kobo.com/v1/products/books/subscriptions",
    "categories": "https://storeapi.kobo.com/v1/categories",
    "categories_page": "https://store.kobobooks.com/ebooks/categories",
    "category": "https://storeapi.kobo.com/v1/categories/{CategoryId}",
    "category_featured_lists": "https://storeapi.kobo.com/v1/categories/{CategoryId}/featured",
    "category_products": "https://storeapi.kobo.com/v1/categories/{CategoryId}/products",
    "checkout_borrowed_book": "https://storeapi.kobo.com/v1/library/borrow",
    "configuration_data": "https://storeapi.kobo.com/v1/configuration",
    "content_access_book": "https://storeapi.kobo.com/v1/products/books/{ProductId}/access",
    "customer_care_live_chat": "https://v2.zopim.com/widget/livechat.html?key=Y6gwUmnu4OATxN3Tli4Av9bYN319BTdO",
    "daily_deal": "https://storeapi.kobo.com/v1/products/dailydeal",
    "deals": "https://storeapi.kobo.com/v1/deals",
    "delete_entitlement": "https://storeapi.kobo.com/v1/library/{Ids}",
    "delete_tag": "https://storeapi.kobo.com/v1/library/tags/{TagId}",
    "delete_tag_items": "https://storeapi.kobo.com/v1/library/tags/{TagId}/items/delete",
    "device_auth": "https://storeapi.kobo.com/v1/auth/device",
    "device_refresh": "https://storeapi.kobo.com/v1/auth/refresh",
    "dictionary_host": "https://kbdownload1-a.akamaihd.net",
    "discovery_host": "https://discovery.kobobooks.com",
    "eula_page": "https://www.kobo.com/termsofuse?style=onestore",
    "exchange_auth": "https://storeapi.kobo.com/v1/auth/exchange",
    "external_book": "https://storeapi.kobo.com/v1/products/books/external/{Ids}",
    "facebook_sso_page": "https://authorize.kobo.com/signin/provider/Facebook/login?returnUrl=http://store.kobobooks.com/",
    "featured_list": "https://storeapi.kobo.com/v1/products/featured/{FeaturedListId}",
    "featured_lists": "https://storeapi.kobo.com/v1/products/featured",
    "free_books_page": {
      "EN": "https://www.kobo.com/{region}/{language}/p/free-ebooks",
      "FR": "https://www.kobo.com/{region}/{language}/p/livres-gratuits",
      "IT": "https://www.kobo.com/{region}/{language}/p/libri-gratuiti",
      "NL": "https://www.kobo.com/{region}/{language}/List/bekijk-het-overzicht-van-gratis-ebooks/QpkkVWnUw8sxmgjSlCbJRg",
      "PT": "https://www.kobo.com/{region}/{language}/p/livros-gratis"
    },
    "fte_feedback": "https://storeapi.kobo.com/v1/products/ftefeedback",
    "get_tests_request": "https://storeapi.kobo.com/v1/analytics/gettests",
    "giftcard_epd_redeem_url": "https://www.kobo.com/{storefront}/{language}/redeem-ereader",
    "giftcard_redeem_url": "https://www.kobo.com/{storefront}/{language}/redeem",
    "help_page": "https://www.kobo.com/help",
    "image_host": "https://my.domain.com/kobo",
    "image_url_quality_template": "https://my.domain.com/kobo/kobo/my-token/{ImageId}/{width}/{height}/{Quality}/isGreyscale/image.jpg",
    "image_url_template": "https://my.domain.com/kobo/kobo/my-token/{ImageId}/{width}/{height}/false/image.jpg",
    "kobo_audiobooks_enabled": "False",
    "kobo_audiobooks_orange_deal_enabled": "False",
    "kobo_audiobooks_subscriptions_enabled": "False",
    "kobo_nativeborrow_enabled": "True",
    "kobo_onestorelibrary_enabled": "False",
    "kobo_redeem_enabled": "True",
    "kobo_shelfie_enabled": "False",
    "kobo_subscriptions_enabled": "False",
    "kobo_superpoints_enabled": "False",
    "kobo_wishlist_enabled": "True",
    "library_book": "https://storeapi.kobo.com/v1/user/library/books/{LibraryItemId}",
    "library_items": "https://storeapi.kobo.com/v1/user/library",
    "library_metadata": "https://storeapi.kobo.com/v1/library/{Ids}/metadata",
    "library_prices": "https://storeapi.kobo.com/v1/user/library/previews/prices",
    "library_stack": "https://storeapi.kobo.com/v1/user/library/stacks/{LibraryItemId}",
    "library_sync": "https://storeapi.kobo.com/v1/library/sync",
    "love_dashboard_page": "https://store.kobobooks.com/{culture}/kobosuperpoints",
    "love_points_redemption_page": "https://store.kobobooks.com/{culture}/KoboSuperPointsRedemption?productId={ProductId}",
    "magazine_landing_page": "https://store.kobobooks.com/emagazines",
    "notifications_registration_issue": "https://storeapi.kobo.com/v1/notifications/registration",
    "oauth_host": "https://oauth.kobo.com",
    "overdrive_account": "https://auth.overdrive.com/account",
    "overdrive_library": "https://{libraryKey}.auth.overdrive.com/library",
    "overdrive_library_finder_host": "https://libraryfinder.api.overdrive.com",
    "overdrive_thunder_host": "https://thunder.api.overdrive.com",
    "password_retrieval_page": "https://www.kobobooks.com/passwordretrieval.html",
    "post_analytics_event": "https://storeapi.kobo.com/v1/analytics/event",
    "privacy_page": "https://www.kobo.com/privacypolicy?style=onestore",
    "product_nextread": "https://storeapi.kobo.com/v1/products/{ProductIds}/nextread",
    "product_prices": "https://storeapi.kobo.com/v1/products/{ProductIds}/prices",
    "product_recommendations": "https://storeapi.kobo.com/v1/products/{ProductId}/recommendations",
    "product_reviews": "https://storeapi.kobo.com/v1/products/{ProductIds}/reviews",
    "products": "https://storeapi.kobo.com/v1/products",
    "provider_external_sign_in_page": "https://authorize.kobo.com/ExternalSignIn/{providerName}?returnUrl=http://store.kobobooks.com/",
    "purchase_buy": "https://www.kobo.com/checkout/createpurchase/",
    "purchase_buy_templated": "https://www.kobo.com/{culture}/checkout/createpurchase/{ProductId}",
    "quickbuy_checkout": "https://storeapi.kobo.com/v1/store/quickbuy/{PurchaseId}/checkout",
    "quickbuy_create": "https://storeapi.kobo.com/v1/store/quickbuy/purchase",
    "rating": "https://storeapi.kobo.com/v1/products/{ProductId}/rating/{Rating}",
    "reading_state": "https://storeapi.kobo.com/v1/library/{Ids}/state",
    "redeem_interstitial_page": "https://store.kobobooks.com",
    "registration_page": "https://authorize.kobo.com/signup?returnUrl=http://store.kobobooks.com/",
    "related_items": "https://storeapi.kobo.com/v1/products/{Id}/related",
    "remaining_book_series": "https://storeapi.kobo.com/v1/products/books/series/{SeriesId}",
    "rename_tag": "https://storeapi.kobo.com/v1/library/tags/{TagId}",
    "review": "https://storeapi.kobo.com/v1/products/reviews/{ReviewId}",
    "review_sentiment": "https://storeapi.kobo.com/v1/products/reviews/{ReviewId}/sentiment/{Sentiment}",
    "shelfie_recommendations": "https://storeapi.kobo.com/v1/user/recommendations/shelfie",
    "sign_in_page": "https://authorize.kobo.com/signin?returnUrl=http://store.kobobooks.com/",
    "social_authorization_host": "https://social.kobobooks.com:8443",
    "social_host": "https://social.kobobooks.com",
    "stacks_host_productId": "https://store.kobobooks.com/collections/byproductid/",
    "store_home": "www.kobo.com/{region}/{language}",
    "store_host": "store.kobobooks.com",
    "store_newreleases": "https://store.kobobooks.com/{culture}/List/new-releases/961XUjtsU0qxkFItWOutGA",
    "store_search": "https://store.kobobooks.com/{culture}/Search?Query={query}",
    "store_top50": "https://store.kobobooks.com/{culture}/ebooks/Top",
    "tag_items": "https://storeapi.kobo.com/v1/library/tags/{TagId}/Items",
    "tags": "https://storeapi.kobo.com/v1/library/tags",
    "taste_profile": "https://storeapi.kobo.com/v1/products/tasteprofile",
    "update_accessibility_to_preview": "https://storeapi.kobo.com/v1/library/{EntitlementIds}/preview",
    "use_one_store": "False",
    "user_loyalty_benefits": "https://storeapi.kobo.com/v1/user/loyalty/benefits",
    "user_platform": "https://storeapi.kobo.com/v1/user/platform",
    "user_profile": "https://storeapi.kobo.com/v1/user/profile",
    "user_ratings": "https://storeapi.kobo.com/v1/user/ratings",
    "user_recommendations": "https://storeapi.kobo.com/v1/user/recommendations",
    "user_reviews": "https://storeapi.kobo.com/v1/user/reviews",
    "user_wishlist": "https://storeapi.kobo.com/v1/user/wishlist",
    "userguide_host": "https://kbdownload1-a.akamaihd.net",
    "wishlist_page": "https://store.kobobooks.com/{region}/{language}/account/wishlist"
  }
}

Screenshot_2023-09-29-20-30-37-08_40deb401b9ffe8e1df2f1cc5ba480b12.jpg

Edit

Investigating a bit more, the kobo endpoint returns a cloudfare block page with the following:

<div class="cf-section cf-wrapper">
        <div class="cf-columns two">
          <div class="cf-column">
            <h2 data-translate="blocked_why_headline">Why have I been blocked?</h2>

            <p data-translate="blocked_why_detail">This website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data.</p>
          </div>

          <div class="cf-column">
            <h2 data-translate="blocked_resolve_headline">What can I do to resolve this?</h2>

            <p data-translate="blocked_resolve_detail">You can email the site owner to let them know you were blocked. Please include what you were doing when this page came up and the Cloudflare Ray ID found at the bottom of this page.</p>
          </div>
        </div>

So it seems that proxied requests are getting blocked, now the question is, is it my google instance getting blocked? Or my current wifi?

Any help would be appreciated. Thanks in advance.

OzzieIsaacs commented 1 year ago

The error message is only triggered in calibre-web as response to a request to "..../v1/initialization" and it's triggered in this block. https://github.com/janeczku/calibre-web/blob/3d07efbb4f67cd43c9dd7fd6f06b4499a40773ac/cps/kobo.py#L1047

The only thing calibre-web does it's requesting the kobo store from the server side parsing the response and then forwarding the answer to your computer. If you get this error message then the response from the kobo store can't be parsed, which leads me to the conclusion that the google instance is blocked. If your wifi (your connection with the google instance) get's blocked, then your computer should show the cloudflare page

jfmlima commented 7 months ago

Closing this as it as my requests were being blocked by cloudflared due to my VM instance in google being tagged.

angelaambroz commented 5 months ago

I'm running into this, running Calibre-web from my local NAS and getting blocked by the Kobo store. Would be great if this could fail without blocking the rest of the local sync, though.