Scrapper error - via Docker

[Fannangir]

Hi, Just ran the docker container, the output shows the log below. Any clue what could be wrong?

[2024-01-05 21:14:50.292] INFO (19): Starting Israeli Bank Firefly iii Importer version: "1.3.7" [2024-01-05 21:14:50.293] INFO (19): Getting state from firefly... [2024-01-05 21:14:50.486] INFO (19): Getting scrap data... [2024-01-05 21:14:52.255] ERROR (19): Unexpected error while scrapping options: { "companyId": "leumi", "startDate": "2019-01-05T21:14:50.487Z", "args": [], "combineInstallments": false, "showBrowser": false, "outputData": { "enableTransactionsFilterByDate": false }, "additionalTransactionInformation": true } error: {} [2024-01-05 21:14:52.260] ERROR (19): Unexpected error while scrapping options: { "companyId": "isracard", "startDate": "2019-01-05T21:14:50.488Z", "args": [], "combineInstallments": false, "showBrowser": false, "outputData": { "enableTransactionsFilterByDate": false }, "additionalTransactionInformation": true } error: {} [2024-01-05 21:14:52.272] ERROR (19): Unexpected error while scrapping options: { "companyId": "visaCal", "startDate": "2019-01-05T21:14:50.488Z", "args": [], "combineInstallments": false, "showBrowser": false, "outputData": { "enableTransactionsFilterByDate": false }, "additionalTransactionInformation": true } error: {} [2024-01-05 21:14:52.279] ERROR (19): Unexpected error while scrapping options: { "companyId": "max", "startDate": "2019-01-05T21:14:50.488Z", "args": [], "combineInstallments": false, "showBrowser": false, "outputData": { "enableTransactionsFilterByDate": false }, "additionalTransactionInformation": true } error: {} [2024-01-05 21:14:52.279] ERROR (19): leumi failed with type GENERAL_ERROR: Failed to launch the browser process! [0105/211450.666604:FATAL:zygote_host_impl_linux.cc(117)] No usable sandbox! Update your kernel or see https://chromium.googlesource.com/chromium/src/+/master/docs/linux/suid_sandbox_development.md for more information on developing with the SUID sandbox. If you want to live dangerously and need an immediate workaround, you can try using --no-sandbox.

0 0x5619d10ae7b9 base::debug::CollectStackTrace()

1 0x5619d101a813 base::debug::StackTrace::StackTrace()

2 0x5619d102b9a6 logging::LogMessage::~LogMessage()

3 0x5619cfa2b41e content::ZygoteHostImpl::Init()

4 0x5619d0fba40e content::ContentMainRunnerImpl::Initialize()

5 0x5619d0fb85dc content::RunContentProcess()

6 0x5619d0fb871c content::ContentMain()

7 0x5619d1014714 headless::(anonymous namespace)::RunContentMain()

8 0x5619d1014408 headless::HeadlessShellMain()

9 0x5619ce428385 ChromeMain

10 0x7f0d6861809b __libc_start_main

11 0x5619ce4281aa _start

Received signal 6

0 0x5619d10ae7b9 base::debug::CollectStackTrace()

1 0x5619d101a813 base::debug::StackTrace::StackTrace()

2 0x5619d10ae361 base::debug::(anonymous namespace)::StackDumpSignalHandler()

3 0x7f0d6a6cb730 (/lib/x86_64-linux-gnu/libpthread-2.28.so+0x1272f)

4 0x7f0d6862b8eb gsignal

5 0x7f0d68616535 abort

6 0x5619d10ad2f5 base::debug::BreakDebugger()

7 0x5619d102be0d logging::LogMessage::~LogMessage()

8 0x5619cfa2b41e content::ZygoteHostImpl::Init()

9 0x5619d0fba40e content::ContentMainRunnerImpl::Initialize()

10 0x5619d0fb85dc content::RunContentProcess()

11 0x5619d0fb871c content::ContentMain()

12 0x5619d1014714 headless::(anonymous namespace)::RunContentMain()

13 0x5619d1014408 headless::HeadlessShellMain()

14 0x5619ce428385 ChromeMain

15 0x7f0d6861809b __libc_start_main

16 0x5619ce4281aa _start

r8: 0000000000000000 r9: 00007ffde5146950 r10: 0000000000000008 r11: 0000000000000246 r12: 00007ffde5147c88 r13: 00007ffde5147c80 r14: 00007ffde5147c70 r15: 00007ffde5147420 di: 0000000000000002 si: 00007ffde5146950 bp: 00007ffde5146ba0 bx: 0000000000000006 dx: 0000000000000000 ax: 0000000000000000 cx: 00007f0d6862b8eb sp: 00007ffde5146950 ip: 00007f0d6862b8eb efl: 0000000000000246 cgf: 002b000000000033 erf: 0000000000000000 trp: 0000000000000000 msk: 0000000000000000 cr2: 0000000000000000 [end of stack trace] Calling _exit(1). Core file will not be generated. TROUBLESHOOTING: https://github.com/puppeteer/puppeteer/blob/main/docs/troubleshooting.md , isracard failed with type GENERAL_ERROR: Failed to launch the browser process! [0105/211450.690445:FATAL:zygote_host_impl_linux.cc(117)] No usable sandbox! Update your kernel or see https://chromium.googlesource.com/chromium/src/+/master/docs/linux/suid_sandbox_development.md for more information on developing with the SUID sandbox. If you want to live dangerously and need an immediate workaround, you can try using --no-sandbox.

0 0x5642b55cf7b9 base::debug::CollectStackTrace()

1 0x5642b553b813 base::debug::StackTrace::StackTrace()

2 0x5642b554c9a6 logging::LogMessage::~LogMessage()

3 0x5642b3f4c41e content::ZygoteHostImpl::Init()

4 0x5642b54db40e content::ContentMainRunnerImpl::Initialize()

5 0x5642b54d95dc content::RunContentProcess()

6 0x5642b54d971c content::ContentMain()

7 0x5642b5535714 headless::(anonymous namespace)::RunContentMain()

8 0x5642b5535408 headless::HeadlessShellMain()

9 0x5642b2949385 ChromeMain

10 0x7f844d85d09b __libc_start_main

11 0x5642b29491aa _start

Received signal 6

0 0x5642b55cf7b9 base::debug::CollectStackTrace()

1 0x5642b553b813 base::debug::StackTrace::StackTrace()

2 0x5642b55cf361 base::debug::(anonymous namespace)::StackDumpSignalHandler()

3 0x7f844f910730 (/lib/x86_64-linux-gnu/libpthread-2.28.so+0x1272f)

4 0x7f844d8708eb gsignal

5 0x7f844d85b535 abort

6 0x5642b55ce2f5 base::debug::BreakDebugger()

7 0x5642b554ce0d logging::LogMessage::~LogMessage()

8 0x5642b3f4c41e content::ZygoteHostImpl::Init()

9 0x5642b54db40e content::ContentMainRunnerImpl::Initialize()

10 0x5642b54d95dc content::RunContentProcess()

11 0x5642b54d971c content::ContentMain()

12 0x5642b5535714 headless::(anonymous namespace)::RunContentMain()

13 0x5642b5535408 headless::HeadlessShellMain()

14 0x5642b2949385 ChromeMain

15 0x7f844d85d09b __libc_start_main

16 0x5642b29491aa _start

r8: 0000000000000000 r9: 00007ffc15294540 r10: 0000000000000008 r11: 0000000000000246 r12: 00007ffc15295878 r13: 00007ffc15295870 r14: 00007ffc15295860 r15: 00007ffc15295010 di: 0000000000000002 si: 00007ffc15294540 bp: 00007ffc15294790 bx: 0000000000000006 dx: 0000000000000000 ax: 0000000000000000 cx: 00007f844d8708eb sp: 00007ffc15294540 ip: 00007f844d8708eb efl: 0000000000000246 cgf: 002b000000000033 erf: 0000000000000000 trp: 0000000000000000 msk: 0000000000000000 cr2: 0000000000000000 [end of stack trace] Calling _exit(1). Core file will not be generated. TROUBLESHOOTING: https://github.com/puppeteer/puppeteer/blob/main/docs/troubleshooting.md , visaCal failed with type GENERAL_ERROR: Failed to launch the browser process! [0105/211450.698414:FATAL:zygote_host_impl_linux.cc(117)] No usable sandbox! Update your kernel or see https://chromium.googlesource.com/chromium/src/+/master/docs/linux/suid_sandbox_development.md for more information on developing with the SUID sandbox. If you want to live dangerously and need an immediate workaround, you can try using --no-sandbox.

0 0x55c8c33867b9 base::debug::CollectStackTrace()

1 0x55c8c32f2813 base::debug::StackTrace::StackTrace()

2 0x55c8c33039a6 logging::LogMessage::~LogMessage()

3 0x55c8c1d0341e content::ZygoteHostImpl::Init()

4 0x55c8c329240e content::ContentMainRunnerImpl::Initialize()

5 0x55c8c32905dc content::RunContentProcess()

6 0x55c8c329071c content::ContentMain()

7 0x55c8c32ec714 headless::(anonymous namespace)::RunContentMain()

8 0x55c8c32ec408 headless::HeadlessShellMain()

9 0x55c8c0700385 ChromeMain

10 0x7fa4de9ca09b __libc_start_main

11 0x55c8c07001aa _start

Received signal 6

0 0x55c8c33867b9 base::debug::CollectStackTrace()

1 0x55c8c32f2813 base::debug::StackTrace::StackTrace()

2 0x55c8c3386361 base::debug::(anonymous namespace)::StackDumpSignalHandler()

3 0x7fa4e0a7d730 (/lib/x86_64-linux-gnu/libpthread-2.28.so+0x1272f)

4 0x7fa4de9dd8eb gsignal

5 0x7fa4de9c8535 abort

6 0x55c8c33852f5 base::debug::BreakDebugger()

7 0x55c8c3303e0d logging::LogMessage::~LogMessage()

8 0x55c8c1d0341e content::ZygoteHostImpl::Init()

9 0x55c8c329240e content::ContentMainRunnerImpl::Initialize()

10 0x55c8c32905dc content::RunContentProcess()

11 0x55c8c329071c content::ContentMain()

12 0x55c8c32ec714 headless::(anonymous namespace)::RunContentMain()

13 0x55c8c32ec408 headless::HeadlessShellMain()

14 0x55c8c0700385 ChromeMain

15 0x7fa4de9ca09b __libc_start_main

16 0x55c8c07001aa _start

r8: 0000000000000000 r9: 00007ffcfdc68140 r10: 0000000000000008 r11: 0000000000000246 r12: 00007ffcfdc69478 r13: 00007ffcfdc69470 r14: 00007ffcfdc69460 r15: 00007ffcfdc68c10 di: 0000000000000002 si: 00007ffcfdc68140 bp: 00007ffcfdc68390 bx: 0000000000000006 dx: 0000000000000000 ax: 0000000000000000 cx: 00007fa4de9dd8eb sp: 00007ffcfdc68140 ip: 00007fa4de9dd8eb efl: 0000000000000246 cgf: 002b000000000033 erf: 0000000000000000 trp: 0000000000000000 msk: 0000000000000000 cr2: 0000000000000000 [end of stack trace] Calling _exit(1). Core file will not be generated. TROUBLESHOOTING: https://github.com/puppeteer/puppeteer/blob/main/docs/troubleshooting.md , max failed with type GENERAL_ERROR: Failed to launch the browser process! [0105/211450.713557:FATAL:zygote_host_impl_linux.cc(117)] No usable sandbox! Update your kernel or see https://chromium.googlesource.com/chromium/src/+/master/docs/linux/suid_sandbox_development.md for more information on developing with the SUID sandbox. If you want to live dangerously and need an immediate workaround, you can try using --no-sandbox.

0 0x55beb5dec7b9 base::debug::CollectStackTrace()

1 0x55beb5d58813 base::debug::StackTrace::StackTrace()

2 0x55beb5d699a6 logging::LogMessage::~LogMessage()

3 0x55beb476941e content::ZygoteHostImpl::Init()

4 0x55beb5cf840e content::ContentMainRunnerImpl::Initialize()

5 0x55beb5cf65dc content::RunContentProcess()

6 0x55beb5cf671c content::ContentMain()

7 0x55beb5d52714 headless::(anonymous namespace)::RunContentMain()

8 0x55beb5d52408 headless::HeadlessShellMain()

9 0x55beb3166385 ChromeMain

10 0x7f343795309b __libc_start_main

11 0x55beb31661aa _start

Received signal 6

0 0x55beb5dec7b9 base::debug::CollectStackTrace()

1 0x55beb5d58813 base::debug::StackTrace::StackTrace()

2 0x55beb5dec361 base::debug::(anonymous namespace)::StackDumpSignalHandler()

3 0x7f3439a06730 (/lib/x86_64-linux-gnu/libpthread-2.28.so+0x1272f)

4 0x7f34379668eb gsignal

5 0x7f3437951535 abort

6 0x55beb5deb2f5 base::debug::BreakDebugger()

7 0x55beb5d69e0d logging::LogMessage::~LogMessage()

8 0x55beb476941e content::ZygoteHostImpl::Init()

9 0x55beb5cf840e content::ContentMainRunnerImpl::Initialize()

10 0x55beb5cf65dc content::RunContentProcess()

11 0x55beb5cf671c content::ContentMain()

12 0x55beb5d52714 headless::(anonymous namespace)::RunContentMain()

13 0x55beb5d52408 headless::HeadlessShellMain()

14 0x55beb3166385 ChromeMain

15 0x7f343795309b __libc_start_main

16 0x55beb31661aa _start

r8: 0000000000000000 r9: 00007ffd168ad850 r10: 0000000000000008 r11: 0000000000000246 r12: 00007ffd168aeb88 r13: 00007ffd168aeb80 r14: 00007ffd168aeb70 r15: 00007ffd168ae320 di: 0000000000000002 si: 00007ffd168ad850 bp: 00007ffd168adaa0 bx: 0000000000000006 dx: 0000000000000000 ax: 0000000000000000 cx: 00007f34379668eb sp: 00007ffd168ad850 ip: 00007f34379668eb efl: 0000000000000246 cgf: 002b000000000033 erf: 0000000000000000 trp: 0000000000000000 msk: 0000000000000000 cr2: 0000000000000000 [end of stack trace] Calling _exit(1). Core file will not be generated. TROUBLESHOOTING: https://github.com/puppeteer/puppeteer/blob/main/docs/troubleshooting.md [2024-01-05 21:14:52.279] INFO (19): Getting or creating accounts... [2024-01-05 21:14:52.340] INFO (19): Manipulating... [2024-01-05 21:14:52.342] INFO (19): Getting firefly transactions to compare... since: "2024-01-04" [2024-01-05 21:14:52.392] INFO (19): Creating transactions to firefly...

count: 0 [2024-01-05 21:14:52.392] INFO (19): Updating transactions types to firefly...

count: 0 [2024-01-05 21:14:52.448] INFO (19): Updating last import... [2024-01-05 21:14:52.483] INFO (19): Done. [2024-01-05 21:14:52.483] INFO (19): Running with cron cron: " 0 0 8 *"

[itairaz1]

Thanks for reporting What is your system/OS?

[Fannangir]

Ubuntu LTS 22.04 as a VM in Proxmox.

[itairaz1]

amd or arm? Anyway, this is issue in Puppeteer, and you can look for troubleshooting at https://github.com/puppeteer/puppeteer/blob/main/docs/troubleshooting.md Suggestion from the log:

No usable sandbox! Update your kernel or see https://chromium.googlesource.com/chromium/src/+/master/docs/linux/suid_sandbox_development.md for more information on developing with the SUID sandbox. If you want to live dangerously and need an immediate workaround, you can try using --no-sandbox.

In case you want to try and running with no-sandbox flag (on your own risk), you can put in the config file (by level):

scraper:
  args:
    - '--no-sandbox'

[Fannangir]

Intel N100. I have added your suggestion to the config.yaml as follows: Unfortunately no change.

[itairaz1]

I think there is a mistake and it should be

scraper:
  options:
    args:
      - '--no-sandbox'

(Another level of options between scraper and args) Please try it as well

[Fannangir]

That did the trick, thank you very much. A new error popped up: [2024-01-07 08:00:47.338] ERROR (25): leumi failed with type GENERIC: waiting for selector #enter_your_account a failed: timeout 30000ms exceeded, max failed with type GENERIC: Navigation timeout of 30000 ms exceeded. As your example is using leumi bank, is there a chance you have seen the error and have a quick solution? I didn't find a solution here: https://github.com/eshaham/israeli-bank-scrapers/issues.

[itairaz1]

AFAIK, this issue is fixed (by israeli-bank-scrapers), and I bumped to the fixed version. But currently there is an issue with releasing (issue with building docker) that I'm currently working on. Hope I will finish it this weekend

[Fannangir]

Thank you very much.

[itairaz1]

I released successfully (1.4.0), but it seemed there is still issue with Leumi, I gave a report here

[Fannangir]

Thank you very much. This is what I have meant in my previous post. I have seen the same topic open with no solution for a while. Hope your info will push for a solution.

[Fannangir]

I saw that the PR was accepted. Could you please update?

[Fannangir]

Thank you very much for the update, I can see that the scrapper works. Unfortunately, I can see the same issue I had with the Home Assistant add-on. I can see the credit cards in firefly, but I can't see the Bank Account data (עובר ושב). I have tried to re-install firefly, although I believe this is not the issue, as the credit cards are all updated. Any chance you have an idea what should I check?

Log from the container (1.4.1): [2024-02-05 08:00:00.649] INFO (19): Getting state from firefly... [2024-02-05 08:00:05.559] INFO (19): Getting scrap data... [2024-02-05 08:01:06.256] INFO (19): Getting or creating accounts... [2024-02-05 08:01:06.516] INFO (19): Manipulating... [2024-02-05 08:01:06.522] INFO (19): Getting firefly transactions to compare... since: "2023-06-11" [2024-02-05 08:01:09.821] INFO (19): Creating transactions to firefly...

count: 3 [2024-02-05 08:01:10.555] INFO (19): Updating transactions types to firefly...

count: 0 [2024-02-05 08:01:10.838] INFO (19): Updating last import... [2024-02-05 08:01:10.890] INFO (19): Done.

[itairaz1]

What do you mean by the same issue? I don't think it related to Docker issue any more Do you mean the same issue that happened to you in the past? did you report it here? If no, please open different ticket and provide sanitized debug log data (remove sensitive data)

[Fannangir]

Thank you very much for the quick action. I have opened it under the hass addon, as I was using it in the past as an addon. I have switched to docker with a thought that it might solve the initial issue. Unfortunately the result is the same. The issue is open here, with all the past logs and the steps you have proposed. Open issue: https://github.com/itairaz1/israeli-bank-firefly-importer-hass-addon/issues/1

would you like me to create a new issue, or just continue the one already open?

[Fannangir]

Hi, Is there a chance you have reviewed the issue?

[Fannangir]

@itairaz1 is there a chance to fix the issue?

[itairaz1]

Hi @Fannangir , Which issue are you talking? The ability to reset?

[Fannangir]

The issue is that the assets include credit card data only (including Leumi card, which is a bank card). Over vashav does not show in firefly. Meaning, it creates the credit cards and updates correctly, but does not enter the bank data.