search

jbms / finance-dl

Tools for automatically downloading/scraping personal financial data.
GNU General Public License v2.0
284 stars 38 forks source link

Amazon(.de) not downloading orders #90

Open ikwtif opened 8 months ago

ikwtif commented 8 months ago

Trying to download amazon(.de) orders but doesn't seem to want to download anything, not sure what condition it was waiting on.

Also had to change orderFilterto timeFilter here to get it to scrape the orders https://github.com/jbms/finance-dl/blob/4b8e28a29b8f0faf5ab3457b5cded2079e73f3fd/finance_dl/amazon.py#L449C24-L449C24

DevTools listening on ws://127.0.0.1:64033/devtools/browser/d936fdb9-bceb-4579-bd14-74d94bfbbda2
 --connect=http://localhost:64028 --session-id=c1a93491b891b70821a27455a8cb60f8
2023-10-29 21:31:05,228 amazon.py:277 [INFO] Initiating log in
2023-10-29 21:31:05,972 amazon.py:284 [INFO] You must be already logged in!
2023-10-29 21:31:06,464 amazon.py:464 [INFO] Retrieving order group: 'den letzten 30 Tagen'
2023-10-29 21:31:06,871 amazon.py:436 [INFO] Found no more pages
2023-10-29 21:31:06,899 amazon.py:464 [INFO] Retrieving order group: 'den letzten 3 Monaten'
2023-10-29 21:31:07,272 amazon.py:436 [INFO] Found no more pages
2023-10-29 21:31:07,306 amazon.py:464 [INFO] Retrieving order group: '2023'
[28972:31576:1029/213110.343:ERROR:device_event_log_impl.cc(225)] [21:31:10.343] USB: usb_service_win.cc:415 Could not read device interface GUIDs: Het systeem kan het opgegeven bestand niet vinden. (0x2)
[28972:31576:1029/213110.344:ERROR:device_event_log_impl.cc(225)] [21:31:10.343] USB: usb_service_win.cc:104 SetupDiGetDeviceProperty({{A45C254E-DF1C-4EFD-8020-67D146A850E0}, 6}) failed: Kan element niet vinden. (0x490)
2023-10-29 21:31:10,385 amazon.py:427 [INFO] Found order '305-6007193-8432350'
2023-10-29 21:31:11,514 amazon.py:427 [INFO] Found order '305-1364969-2322728'
2023-10-29 21:31:12,697 amazon.py:427 [INFO] Found order '305-7507294-1425151'
2023-10-29 21:31:12,701 amazon.py:436 [INFO] Found no more pages
2023-10-29 21:31:12,725 amazon.py:464 [INFO] Retrieving order group: '2022'
2023-10-29 21:31:14,640 amazon.py:427 [INFO] Found order '304-7286585-0240307'
2023-10-29 21:31:16,847 amazon.py:427 [INFO] Found order '304-0532664-5059518'
2023-10-29 21:31:18,000 amazon.py:427 [INFO] Found order '304-9561231-4685903'
2023-10-29 21:31:19,150 amazon.py:427 [INFO] Found order '304-1342413-3851547'
2023-10-29 21:31:20,295 amazon.py:427 [INFO] Found order '304-4127407-3957141'
2023-10-29 21:31:21,428 amazon.py:427 [INFO] Found order '304-9986951-1473951'
2023-10-29 21:31:22,585 amazon.py:427 [INFO] Found order '304-1950788-6290723'
2023-10-29 21:31:23,732 amazon.py:427 [INFO] Found order '304-7040015-3071518'
2023-10-29 21:31:24,874 amazon.py:427 [INFO] Found order '304-6516850-4829932'
2023-10-29 21:31:26,051 amazon.py:427 [INFO] Found order '304-5372088-9945954'
2023-10-29 21:31:26,069 amazon.py:441 [INFO] Next page.
2023-10-29 21:31:28,034 amazon.py:427 [INFO] Found order '304-7060525-9372365'
2023-10-29 21:31:29,158 amazon.py:427 [INFO] Found order '304-4859404-2549141'
2023-10-29 21:31:30,296 amazon.py:427 [INFO] Found order '304-4924192-0299519'
2023-10-29 21:31:31,412 amazon.py:427 [INFO] Found order '305-8808333-1320322'
2023-10-29 21:31:32,559 amazon.py:427 [INFO] Found order '304-0657323-9229902'
2023-10-29 21:31:33,739 amazon.py:427 [INFO] Found order '305-2521019-8087569'
2023-10-29 21:31:34,939 amazon.py:427 [INFO] Found order '306-6714178-4185146'
2023-10-29 21:31:34,945 amazon.py:436 [INFO] Found no more pages
2023-10-29 21:31:34,975 amazon.py:464 [INFO] Retrieving order group: '2021'
2023-10-29 21:31:37,610 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-5183444-7545952'
2023-10-29 21:31:37,674 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-5183444-7545952'
2023-10-29 21:31:37,739 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-5183444-7545952'
2023-10-29 21:31:37,819 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-5183444-7545952'
2023-10-29 21:31:37,882 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-5183444-7545952'
2023-10-29 21:31:37,958 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-5183444-7545952'
2023-10-29 21:31:39,651 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-4252588-7295546'
2023-10-29 21:31:39,709 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-4252588-7295546'
2023-10-29 21:31:39,781 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-4252588-7295546'
2023-10-29 21:31:39,845 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-4252588-7295546'
2023-10-29 21:31:39,869 amazon.py:441 [INFO] Next page.
2023-10-29 21:31:41,930 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-2656410-5349950'
2023-10-29 21:31:42,006 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-2656410-5349950'
2023-10-29 21:31:42,086 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-2656410-5349950'
2023-10-29 21:31:42,151 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-2656410-5349950'
2023-10-29 21:31:42,218 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-2656410-5349950'
2023-10-29 21:31:42,291 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-2656410-5349950'
2023-10-29 21:31:43,989 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-4665213-3963505'
2023-10-29 21:31:44,059 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-4665213-3963505'
2023-10-29 21:31:44,124 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-4665213-3963505'
2023-10-29 21:31:44,194 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-4665213-3963505'
2023-10-29 21:31:44,206 amazon.py:441 [INFO] Next page.
2023-10-29 21:31:46,816 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-6833943-6329939'
2023-10-29 21:31:46,879 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-6833943-6329939'
2023-10-29 21:31:46,953 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-6833943-6329939'
2023-10-29 21:31:47,016 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-6833943-6329939'
2023-10-29 21:31:47,081 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-6833943-6329939'
2023-10-29 21:31:47,146 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-6833943-6329939'
2023-10-29 21:31:48,830 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-4527795-0562718'
2023-10-29 21:31:48,892 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-4527795-0562718'
2023-10-29 21:31:48,970 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-4527795-0562718'
2023-10-29 21:31:49,034 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-4527795-0562718'
2023-10-29 21:31:49,050 amazon.py:441 [INFO] Next page.
2023-10-29 21:31:51,003 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-8824203-0488334'
2023-10-29 21:31:51,059 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-8824203-0488334'
2023-10-29 21:31:51,124 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-8824203-0488334'
2023-10-29 21:31:51,177 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-8824203-0488334'
2023-10-29 21:31:51,241 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-8824203-0488334'
2023-10-29 21:31:51,246 amazon.py:436 [INFO] Found no more pages
2023-10-29 21:31:51,279 amazon.py:464 [INFO] Retrieving order group: '2020'
2023-10-29 21:31:53,301 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-8315787-5112305'
2023-10-29 21:31:53,368 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-8315787-5112305'
2023-10-29 21:31:53,446 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-8315787-5112305'
2023-10-29 21:31:53,510 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-8315787-5112305'
2023-10-29 21:31:53,580 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-8315787-5112305'
2023-10-29 21:31:53,641 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-8315787-5112305'
2023-10-29 21:31:55,319 amazon.py:425 [INFO] Skipping already-downloaded invoice: '302-6574931-4521130'
2023-10-29 21:31:55,385 amazon.py:425 [INFO] Skipping already-downloaded invoice: '302-6574931-4521130'
2023-10-29 21:31:55,451 amazon.py:425 [INFO] Skipping already-downloaded invoice: '302-6574931-4521130'
2023-10-29 21:31:55,510 amazon.py:425 [INFO] Skipping already-downloaded invoice: '302-6574931-4521130'
2023-10-29 21:31:55,527 amazon.py:441 [INFO] Next page.
2023-10-29 21:31:57,390 amazon.py:425 [INFO] Skipping already-downloaded invoice: '302-3192929-2299507'
2023-10-29 21:31:57,444 amazon.py:425 [INFO] Skipping already-downloaded invoice: '302-3192929-2299507'
2023-10-29 21:31:57,499 amazon.py:425 [INFO] Skipping already-downloaded invoice: '302-3192929-2299507'
2023-10-29 21:31:57,556 amazon.py:425 [INFO] Skipping already-downloaded invoice: '302-3192929-2299507'
2023-10-29 21:31:57,607 amazon.py:425 [INFO] Skipping already-downloaded invoice: '302-3192929-2299507'
2023-10-29 21:31:57,659 amazon.py:425 [INFO] Skipping already-downloaded invoice: '302-3192929-2299507'
2023-10-29 21:31:57,710 amazon.py:425 [INFO] Skipping already-downloaded invoice: '302-3192929-2299507'
2023-10-29 21:31:57,716 amazon.py:436 [INFO] Found no more pages
2023-10-29 21:31:57,744 amazon.py:464 [INFO] Retrieving order group: '2019'
2023-10-29 21:31:59,862 amazon.py:425 [INFO] Skipping already-downloaded invoice: '302-6447883-6259537'
2023-10-29 21:31:59,931 amazon.py:425 [INFO] Skipping already-downloaded invoice: '302-6447883-6259537'
2023-10-29 21:31:59,993 amazon.py:425 [INFO] Skipping already-downloaded invoice: '302-6447883-6259537'
2023-10-29 21:32:00,058 amazon.py:425 [INFO] Skipping already-downloaded invoice: '302-6447883-6259537'
2023-10-29 21:32:00,118 amazon.py:425 [INFO] Skipping already-downloaded invoice: '302-6447883-6259537'
2023-10-29 21:32:00,181 amazon.py:425 [INFO] Skipping already-downloaded invoice: '302-6447883-6259537'
2023-10-29 21:32:01,322 amazon.py:425 [INFO] Skipping already-downloaded invoice: '306-7975075-7649159'
2023-10-29 21:32:01,382 amazon.py:425 [INFO] Skipping already-downloaded invoice: '306-7975075-7649159'
2023-10-29 21:32:01,450 amazon.py:425 [INFO] Skipping already-downloaded invoice: '306-7975075-7649159'
2023-10-29 21:32:01,514 amazon.py:425 [INFO] Skipping already-downloaded invoice: '306-7975075-7649159'
2023-10-29 21:32:01,534 amazon.py:441 [INFO] Next page.
2023-10-29 21:32:03,450 amazon.py:425 [INFO] Skipping already-downloaded invoice: '306-6615968-4057139'
2023-10-29 21:32:03,453 amazon.py:436 [INFO] Found no more pages
2023-10-29 21:32:03,487 amazon.py:464 [INFO] Retrieving order group: '2018'
2023-10-29 21:32:05,546 amazon.py:425 [INFO] Skipping already-downloaded invoice: '302-9120740-4793910'
2023-10-29 21:32:05,611 amazon.py:425 [INFO] Skipping already-downloaded invoice: '302-9120740-4793910'
2023-10-29 21:32:05,684 amazon.py:425 [INFO] Skipping already-downloaded invoice: '302-9120740-4793910'
2023-10-29 21:32:05,749 amazon.py:425 [INFO] Skipping already-downloaded invoice: '302-9120740-4793910'
2023-10-29 21:32:05,814 amazon.py:425 [INFO] Skipping already-downloaded invoice: '302-9120740-4793910'
2023-10-29 21:32:05,883 amazon.py:425 [INFO] Skipping already-downloaded invoice: '302-9120740-4793910'
2023-10-29 21:32:07,038 amazon.py:425 [INFO] Skipping already-downloaded invoice: '028-4791616-8929953'
2023-10-29 21:32:07,107 amazon.py:425 [INFO] Skipping already-downloaded invoice: '028-4791616-8929953'
2023-10-29 21:32:07,172 amazon.py:425 [INFO] Skipping already-downloaded invoice: '028-4791616-8929953'
2023-10-29 21:32:07,234 amazon.py:425 [INFO] Skipping already-downloaded invoice: '028-4791616-8929953'
2023-10-29 21:32:07,246 amazon.py:441 [INFO] Next page.
2023-10-29 21:32:09,193 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-7201984-1193117'
2023-10-29 21:32:09,244 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-7201984-1193117'
2023-10-29 21:32:09,301 amazon.py:425 [INFO] Skipping already-downloaded invoice: '305-7201984-1193117'
2023-10-29 21:32:09,308 amazon.py:436 [INFO] Found no more pages
2023-10-29 21:32:09,350 amazon.py:464 [INFO] Retrieving order group: '2017'
2023-10-29 21:32:09,759 amazon.py:436 [INFO] Found no more pages
2023-10-29 21:32:09,783 amazon.py:464 [INFO] Retrieving order group: '2016'
2023-10-29 21:32:10,184 amazon.py:436 [INFO] Found no more pages
2023-10-29 21:32:10,211 amazon.py:464 [INFO] Retrieving order group: '2015'
2023-10-29 21:32:10,612 amazon.py:436 [INFO] Found no more pages
2023-10-29 21:32:10,648 amazon.py:464 [INFO] Retrieving order group: 'Archivierte Bestellungen'
2023-10-29 21:32:11,001 amazon.py:436 [INFO] Found no more pages
Traceback (most recent call last):
  File "D:\Finance\finance-dl-master\finance_dl\scrape_lib.py", line 403, in retry
    return func()
  File "D:\Finance\finance-dl-master\finance_dl\scrape_lib.py", line 423, in fetch
    scraper.run()
  File "D:\Finance\finance-dl-master\finance_dl\amazon.py", line 585, in run
    self.get_orders(
  File "D:\Finance\finance-dl-master\finance_dl\amazon.py", line 478, in get_orders
    retrieve_all_order_groups()
  File "D:\Finance\finance-dl-master\finance_dl\amazon.py", line 448, in retrieve_all_order_groups
    (order_filter,), = self.wait_and_return(
  File "D:\Finance\finance-dl-master\finance_dl\scrape_lib.py", line 239, in wait_and_return
    WebDriverWait(self.driver, timeout).until(predicate, message=message)
  File "D:\Finance\env-financedl\lib\site-packages\selenium\webdriver\support\wait.py", line 87, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: Waiting to match conditions
ikwtif commented 8 months ago

Not sure if it's related, but had this same error on .co.uk when I was still logged into amazon from a previous attempt. I logged out and the error was gone

Traceback (most recent call last):
  File "D:\Finance\finance-dl-master\finance_dl\scrape_lib.py", line 403, in retry
    return func()
  File "D:\Finance\finance-dl-master\finance_dl\scrape_lib.py", line 423, in fetch
    scraper.run()
  File "D:\Finance\finance-dl-master\finance_dl\amazon.py", line 585, in run
    self.get_orders(
  File "D:\Finance\finance-dl-master\finance_dl\amazon.py", line 472, in get_orders
    (orders_link,), = self.wait_and_return(
  File "D:\Finance\finance-dl-master\finance_dl\scrape_lib.py", line 239, in wait_and_return
    WebDriverWait(self.driver, timeout).until(predicate, message=message)
  File "D:\Finance\env-financedl\lib\site-packages\selenium\webdriver\support\wait.py", line 87, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: Waiting to match conditions