Closed johnlescault closed 2 years ago
These are the limitations of the API. A maximum of 1000 entries are sent in one reply from the server. This is not a bug. This is how the Amazon/Audible server works! In theory, you can specify the page=2
param to get the next entries. I'll test it out. If it works, i‘ll add this to audible-cli
.
@johnlescault I've created a pull request wich should work. Can you test these please?
I'm guessing my last comment was in the pull request.
I still can't get the library export to work. As you requested I ran without the timeout option (which never worked for me even using the master version of this - I assumed due to slow internet)
Here is the output I get:
Traceback (most recent call last):
File "/home/joey/.local/lib/python3.8/site-packages/httpx/_transports/default.py", line 62, in map_httpcore_exceptions
yield
File "/home/joey/.local/lib/python3.8/site-packages/httpx/_transports/default.py", line 276, in handle_async_request
) = await self._pool.handle_async_request(
File "/home/joey/.local/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 234, in handle_async_request
response = await connection.handle_async_request(
File "/home/joey/.local/lib/python3.8/site-packages/httpcore/_async/connection.py", line 148, in handle_async_request
return await self.connection.handle_async_request(
File "/home/joey/.local/lib/python3.8/site-packages/httpcore/_async/http11.py", line 128, in handle_async_request
) = await self._receive_response(timeout)
File "/home/joey/.local/lib/python3.8/site-packages/httpcore/_async/http11.py", line 189, in _receive_response
event = await self._receive_event(timeout)
File "/home/joey/.local/lib/python3.8/site-packages/httpcore/_async/http11.py", line 225, in _receive_event
data = await self.socket.read(self.READ_NUM_BYTES, timeout)
File "/home/joey/.local/lib/python3.8/site-packages/httpcore/_backends/anyio.py", line 63, in read
raise ReadTimeout from None
httpcore.ReadTimeout
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/audible/client.py", line 254, in _request
resp = await self.session.request(method, url, **kwargs)
File "/home/joey/.local/lib/python3.8/site-packages/httpx/_client.py", line 1483, in request
response = await self.send(
File "/home/joey/.local/lib/python3.8/site-packages/httpx/_client.py", line 1571, in send
response = await self._send_handling_auth(
File "/home/joey/.local/lib/python3.8/site-packages/httpx/_client.py", line 1599, in _send_handling_auth
response = await self._send_handling_redirects(
File "/home/joey/.local/lib/python3.8/site-packages/httpx/_client.py", line 1636, in _send_handling_redirects
response = await self._send_single_request(request)
File "/home/joey/.local/lib/python3.8/site-packages/httpx/_client.py", line 1673, in _send_single_request
response = await transport.handle_async_request(request)
File "/home/joey/.local/lib/python3.8/site-packages/httpx/_transports/default.py", line 271, in handle_async_request
(
File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
self.gen.throw(type, value, traceback)
File "/home/joey/.local/lib/python3.8/site-packages/httpx/_transports/default.py", line 79, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ReadTimeout
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/audible", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python3.8/dist-packages/audible_cli/cli.py", line 54, in main
sys.exit(cli(*args, **kwargs))
File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/usr/lib/python3/dist-packages/click/decorators.py", line 64, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/audible_cli/cmds/cmd_library.py", line 128, in export_library
loop.run_until_complete(_export_library(session.auth, **params))
File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/usr/local/lib/python3.8/dist-packages/audible_cli/cmds/cmd_library.py", line 59, in _export_library
library = await _get_library(auth, **params)
File "/usr/local/lib/python3.8/dist-packages/audible_cli/cmds/cmd_library.py", line 21, in _get_library
library = await Library.aget_from_api(
File "/usr/local/lib/python3.8/dist-packages/audible_cli/models.py", line 430, in aget_from_api
resp = await fetch_library(request_params)
File "/usr/local/lib/python3.8/dist-packages/audible_cli/models.py", line 408, in fetch_library
r = await api_client.get(
File "/usr/local/lib/python3.8/dist-packages/audible/client.py", line 273, in _request
raise NotResponding
audible.exceptions.NotResponding: API request timed out, please be patient.
Yeah, I've merged the pr in the meantime. I've changed something now. Can you please pull the changes and install the new master branch and test with -t 9000
again. Please try the library list
command too.
I‘ve updated the package again. You can now disable timout with -t 0
option.
Something broke... it times out regardless now.
audible library export -t 0 (times out instantly) audible download --all --aaxc --cover --cover-size 1215 --chapter --pdf --jobs 3 (also times out instantly, but used to work)
Also just wanted to add - great work on this. It definately is the best tool out there for audible!
Thank you. Good to hear that. Does it run now setting -t 9000
?
No - it had also timed out. Same as it did before. Just want to reinforce that I have bad internet though (5mbps). Before you spend too much time trying to troubleshoot this, could this be the issue? I'll be able to further test once I go to my parents, but it's not for a bit as they are 3 hours away.
That's really strange. The manpage for httpx describes that timeout=None
disable the timeout. That's what I have done. Maybe you can pull the changes and reinstall again and than call audible --version
?
I don't think that your internet connection is the issue. You got a ReadTimeout exception. This means, you don't got the full response in time and your client raises the exception. But with disabled timeout, your client should wait infinite. Only the server could disconnect at this moment.
Just reinstalled via: git clone https://github.com/mkb79/audible-cli.git shows version 'audible, version 0.0.7dev1'
Should I be reinstalling the audible package as well and not just audible-cli?
You can try it to update the Audible package as well. Which version of httpx you have installed? It should be 0.20.*!
I did have a lower version actually, had to manually put the new version. This doesn't seem to have corrected either the download portion or library export.
I did get a different error though this time: audible.exceptions.BadRequest: Bad Request (400): Implied library size is unsupported
Which version of httpx
you have installed now?
I believe the one you said. I ran command:
sudo python3 -m pip install "httpx==0.20.*"
I can't find a way to verify the version number from httpx as the --help doesn't have that as an option.. Sorry a little outside my normal zone :)
Please run pip list | grep httpx
Oddly enough it brought up: httpx (1.0.0b0) When I updated audible package it did say the following: Successfully installed audible-0.7.0.dev1 httpcore-0.13.7 httpx-0.20.0
Okay, then run pip uninstall httpx
and then pip install httpx==0.20.0
You can try in addition the binary audible_debian11
in the bin folder.
Ok I did the uninstall and that seems to have progressed me back to the original issue.
download --all works flawlessly just as before.
library export -t 0 brings the following error:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/audible/client.py", line 265, in _request
resp.raise_for_status()
File "/home/joey/.local/lib/python3.8/site-packages/httpx/_models.py", line 1507, in raise_for_status
raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Server error '500 Internal Server Error' for url 'https://api.audible.ca/1.0/library?response_groups=contributors%2C+media%2C+price%2C+product_attrs%2C+product_desc%2C+product_extended_attrs%2C+product_plan_details%2C+product_plans%2C+rating%2C+sample%2C+sku%2C+series%2C+reviews%2C+ws4v%2C+origin%2C+relationships%2C+review_attrs%2C+categories%2C+badge_types%2C+category_ladders%2C+claim_code_url%2C+is_downloaded%2C+is_finished%2C+is_returnable%2C+origin_asin%2C+pdf_url%2C+percent_complete%2C+provided_review&page=2&num_results=1000'
For more information check: https://httpstatuses.com/500
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/audible", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python3.8/dist-packages/audible_cli/cli.py", line 54, in main
sys.exit(cli(*args, **kwargs))
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/click/decorators.py", line 84, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/audible_cli/cmds/cmd_library.py", line 129, in export_library
loop.run_until_complete(_export_library(session.auth, **params))
File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/usr/local/lib/python3.8/dist-packages/audible_cli/cmds/cmd_library.py", line 60, in _export_library
library = await _get_library(auth, **params)
File "/usr/local/lib/python3.8/dist-packages/audible_cli/cmds/cmd_library.py", line 23, in _get_library
library = await Library.aget_from_api(
File "/usr/local/lib/python3.8/dist-packages/audible_cli/models.py", line 428, in aget_from_api
resp = await fetch_library(request_params)
File "/usr/local/lib/python3.8/dist-packages/audible_cli/models.py", line 406, in fetch_library
r = await api_client.get(
File "/usr/local/lib/python3.8/dist-packages/audible/client.py", line 279, in _request
self._raise_for_status_error(exc.response)
File "/usr/local/lib/python3.8/dist-packages/audible/client.py", line 115, in _raise_for_status_error
raise UnexpectedError(resp, data)
audible.exceptions.UnexpectedError: Internal Server Error (500): <html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>We're sorry!</title>
</head>
<body style="padding: 4em; color: black; background-color: white; font-family: Verdana,Arial,Helvetica,sans-serif;">
<h2>We're sorry!</h2>
<p>
An error occurred when we tried to process your request.
Rest assured, we're already working on the problem and expect to resolve it shortly.
</p>
<h2>Désolés!</h2>
<p>
Une erreur s'est produite lorsque nous avons tenté de traiter votre requête.
Soyez assuré que nous travaillons déjà à la résolution du problème que nous pensons trouver très rapidement.
</p>
</body>
</html>
It does seem to fail on page 2 (as per the URL in the error message), and it is not timing out like it was before. It also takes MUCH longer to fail so I think you did fix that issue. Is there a way for the library export to specify the pages you want exported? Maybe we can try to ONLY export page 2 and see what it does?
I've created a new dev branch with some changes. Can you test these branch? If this does not work I will implement a page option.
Didn't read your request perfectly at first so I re-downloaded the master copy and installed it. It seems to have worked this time! I feel like I had already tried the master so not sure why the different result.... The master still needed -t 90000 though
I figured I would try the dev branch once I re-read what you asked, and that doesn't seem to work. audible library export (audible.exceptions.NotResponding: API request timed out, please be patient.) audible library export -t 90000 (Only returned a file with 233 books, along with an error:
Traceback (most recent call last):
File "/home/joey/.local/bin/audible", line 8, in <module>
sys.exit(main())
File "/home/joey/.local/lib/python3.8/site-packages/audible_cli/cli.py", line 54, in main
sys.exit(cli(*args, **kwargs))
File "/home/joey/.local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/home/joey/.local/lib/python3.8/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/home/joey/.local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/joey/.local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/joey/.local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/joey/.local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/home/joey/.local/lib/python3.8/site-packages/click/decorators.py", line 84, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
File "/home/joey/.local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/home/joey/.local/lib/python3.8/site-packages/audible_cli/cmds/cmd_library.py", line 129, in export_library
loop.run_until_complete(_export_library(session.auth, **params))
File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/home/joey/.local/lib/python3.8/site-packages/audible_cli/cmds/cmd_library.py", line 91, in _export_library
data_row["rating"] = v["overall_distribution"][
TypeError: 'NoneType' object is not subscriptable
That are good news. The dev
branch could get your entire library. But the next step, parsing the lib, fails. I know what happens and will debug this.
Had the library which you export with the master branch contains all your books?
I've updated the dev branch now. Can you pull the changes and install with --upgrade
flag of the pip command.
The master branch is actually missing about 14 books - I didn't look close enough before to notice it as the number was very close. I am more than happy with that progress if continuing is too much work for you! I appreciate everything you've done so far!
Please forgive me if I did the update wrong. I wasn't sure how to put the --upgrade as I have been downloading the zip manually and installing it that way, so I went here: https://github.com/mkb79/audible-cli/tree/dev then to code -> download zip and reinstalled over the previous version. Let me know if I did that wrong. I've been learning a lot about github with this issue but still not at expert level... lol
audible library export -> failed again
audible library export -t 90000 Gave me a tsv file with 465 books only and the following error: (from what I can tell the same one as before)
Traceback (most recent call last):
File "/home/joey/.local/bin/audible", line 8, in <module>
sys.exit(main())
File "/home/joey/.local/lib/python3.8/site-packages/audible_cli/cli.py", line 54, in main
sys.exit(cli(*args, **kwargs))
File "/home/joey/.local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/home/joey/.local/lib/python3.8/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/home/joey/.local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/joey/.local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/joey/.local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/joey/.local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/home/joey/.local/lib/python3.8/site-packages/click/decorators.py", line 84, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
File "/home/joey/.local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/home/joey/.local/lib/python3.8/site-packages/audible_cli/cmds/cmd_library.py", line 129, in export_library
loop.run_until_complete(_export_library(session.auth, **params))
File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/home/joey/.local/lib/python3.8/site-packages/audible_cli/cmds/cmd_library.py", line 91, in _export_library
data_row["rating"] = v["overall_distribution"][
TypeError: 'NoneType' object is not subscriptable
There errors above was bugfixed in dev. But your line of code is the old one. Can you check your installation please.
I think I realized how I installed wrong. I am not getting the following error during installation of the dev branch: ERROR: audible-cli 0.0.dev10 has requirement httpx==0.16.*, but you'll have httpx 0.20.0 which is incompatible.
Oh, something is going wrong. Can you delete the audible-cli
folder please and then run
python3 -m pip uninstall audible-cli
git clone --branch dev https://github.com/mkb79/audible-cli
cd audible-cli
python3 -m pip install --upgrade .
Ok, did exactly what you asked for and here are the results:
audible library export file contains 1239 books (missing books). The following error occured:
Traceback (most recent call last):
File "/home/joey/.local/bin/audible", line 8, in <module>
sys.exit(main())
File "/home/joey/.local/lib/python3.8/site-packages/audible_cli/cli.py", line 54, in main
sys.exit(cli(*args, **kwargs))
File "/home/joey/.local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/home/joey/.local/lib/python3.8/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/home/joey/.local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/joey/.local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/joey/.local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/joey/.local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/home/joey/.local/lib/python3.8/site-packages/click/decorators.py", line 84, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
File "/home/joey/.local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/home/joey/.local/lib/python3.8/site-packages/audible_cli/cmds/cmd_library.py", line 129, in export_library
loop.run_until_complete(_export_library(session.auth, **params))
File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/home/joey/.local/lib/python3.8/site-packages/audible_cli/cmds/cmd_library.py", line 91, in _export_library
data_row["rating"] = v.get("overall_distribution", {}).get(
AttributeError: 'NoneType' object has no attribute 'get'
I believe the same error occured for "audible library export -t 90000"
Traceback (most recent call last):
File "/home/joey/.local/bin/audible", line 8, in <module>
sys.exit(main())
File "/home/joey/.local/lib/python3.8/site-packages/audible_cli/cli.py", line 54, in main
sys.exit(cli(*args, **kwargs))
File "/home/joey/.local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/home/joey/.local/lib/python3.8/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/home/joey/.local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/joey/.local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/joey/.local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/joey/.local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/home/joey/.local/lib/python3.8/site-packages/click/decorators.py", line 84, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
File "/home/joey/.local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/home/joey/.local/lib/python3.8/site-packages/audible_cli/cmds/cmd_library.py", line 129, in export_library
loop.run_until_complete(_export_library(session.auth, **params))
File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/home/joey/.local/lib/python3.8/site-packages/audible_cli/cmds/cmd_library.py", line 91, in _export_library
data_row["rating"] = v.get("overall_distribution", {}).get(
AttributeError: 'NoneType' object has no attribute 'get'
Not sure if it matters, it actually returned 1238 books (first line is a header)
Okay, this means parsing book 1239 gives a error. There must be something with these book. Will take a look on this.
Are you able to make it Echo the title of book 1238 and I could tell you what book 1239 is?
Or wait, scratch that. Its in the export. Let me find it.
Oh, something is going wrong. Can you delete the
audible-cli
folder please and then runpython3 -m pip uninstall audible-cli git clone --branch dev https://github.com/mkb79/audible-cli cd audible-cli python3 -m pip install --upgrade .
Okay, I think I got it. Can you delete the files and reinstall the package how I described above.
It worked!!!! Thank you so much!
No problem at all. I've increased the number of results per request to the maximum of 1000 entries. This results in less requests. Can you pull, reinstall and test the package again, please?
It still works! If possible and if I'm not asking too much after all this, would it be possible to add a visual when you download of what book you are at? I could just see that it would be in the dark while downloading large files. Something like "Downloading Book 1 of 1500" Thanks again!
You mean when you use the download command? Or the library command? When you download a book, it should actually show you the current book and a progress bar.
For the download command. When using download -all specifically (with a large library like mine). There is a progress bar for each individual book, but I have no way of knowing how many more books are left to download. Mix that with slow internet connection, and it is really useful of guaging an approximate of how long is left :)
I'm probably the only one with this issue though :)
Yeah, I can implement this in future releases!
Describe the bug I have a library of 1357 books. It seems to cut off at 1000 books, missing the last 357. It seems to do the 1000 "newest added" books. I am using this mixed with the audible-cli git. I realize I have an abnormally large library, but more the reason I'd like to backup my purchases.
To Reproduce In terminal, either: audible download all [...] OR audible library export
Expected behavior Would like to request for support of greater than 1000 books
Desktop (please complete the following information):
Thanks in advanced - really happy with this tool otherwise!