Failed to connect to fireflyiii API on start

[myakove]

Once start it fail to connect to fireflyiii API (Running both from docker)

from the log:

israeli-bank-firefly-importer    | 
israeli-bank-firefly-importer    | > israeli-bank-firefly-importer@0.6.5 start
israeli-bank-firefly-importer    | > node src/index.js
israeli-bank-firefly-importer    | 
israeli-bank-firefly-importer    | [2023-02-13 10:27:37.777] INFO (18): Getting state from firefly...
israeli-bank-firefly-importer    | [2023-02-13 10:27:38.157] ERROR (18): Fatal error
israeli-bank-firefly-importer    |     err: {
israeli-bank-firefly-importer    |       "message": "Request failed with status code 404",
israeli-bank-firefly-importer    |       "name": "AxiosError",
israeli-bank-firefly-importer    |       "stack":
israeli-bank-firefly-importer    |           AxiosError: Request failed with status code 404
israeli-bank-firefly-importer    |               at settle (file:///app/node_modules/axios/lib/core/settle.js:19:12)
israeli-bank-firefly-importer    |               at IncomingMessage.handleStreamEnd (file:///app/node_modules/axios/lib/adapters/http.js:548:11)
israeli-bank-firefly-importer    |               at IncomingMessage.emit (node:events:524:35)
israeli-bank-firefly-importer    |               at endReadableNT (node:internal/streams/readable:1359:12)
israeli-bank-firefly-importer    |               at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
israeli-bank-firefly-importer    |       "config": {
israeli-bank-firefly-importer    |         "transitional": {
israeli-bank-firefly-importer    |           "silentJSONParsing": true,
israeli-bank-firefly-importer    |           "forcedJSONParsing": true,
israeli-bank-firefly-importer    |           "clarifyTimeoutError": false
israeli-bank-firefly-importer    |         },
israeli-bank-firefly-importer    |         "adapter": [
israeli-bank-firefly-importer    |           "xhr",
israeli-bank-firefly-importer    |           "http"
israeli-bank-firefly-importer    |         ],
israeli-bank-firefly-importer    |         "transformRequest": [
israeli-bank-firefly-importer    |           null
israeli-bank-firefly-importer    |         ],
israeli-bank-firefly-importer    |         "transformResponse": [
israeli-bank-firefly-importer    |           null
israeli-bank-firefly-importer    |         ],
israeli-bank-firefly-importer    |         "timeout": 0,
israeli-bank-firefly-importer    |         "xsrfCookieName": "XSRF-TOKEN",
israeli-bank-firefly-importer    |         "xsrfHeaderName": "X-XSRF-TOKEN",
israeli-bank-firefly-importer    |         "maxContentLength": -1,
israeli-bank-firefly-importer    |         "maxBodyLength": -1,
israeli-bank-firefly-importer    |         "env": {},
israeli-bank-firefly-importer    |         "headers": {
israeli-bank-firefly-importer    |           "Accept": "application/json, text/plain, */*",
israeli-bank-firefly-importer    |           "Authorization": "Bearer <TOKEN REMOVED>",
israeli-bank-firefly-importer    |           "User-Agent": "axios/1.3.2",
israeli-bank-firefly-importer    |           "Accept-Encoding": "gzip, compress, deflate, br"
israeli-bank-firefly-importer    |         },
israeli-bank-firefly-importer    |         "baseURL": "http://<fireflyiii container IP>:8089",
israeli-bank-firefly-importer    |         "method": "get",
israeli-bank-firefly-importer    |         "url": "/api/v1/preferences/israeli-bank-importer"
israeli-bank-firefly-importer    |       },
israeli-bank-firefly-importer    |       "code": "ERR_BAD_REQUEST",
israeli-bank-firefly-importer    |       "status": 404
israeli-bank-firefly-importer    |     }

From fireflyiii log:

fireflyiii                       | 172.18.0.1 - - [13/Feb/2023:10:27:37 +0200] "GET /api/v1/preferences/israeli-bank-importer HTTP/1.1" 404 1050 "-" "axios/1.3.2"

israeli-bank-importer is not listed under https://api-docs.firefly-iii.org/#/preferences Maybe I'm missing a step in the installation?

I have no problem to execute:

curl -X GET 'http://<fireflyiii container IP>:8089/api/v1/preferences/currencyPreference' \
  -H 'accept: application/vnd.api+json' \
  -H 'Authorization: Bearer ' <TOKEN REMOVED>'\
  -H 'Content-Type: application/json'

[myakove]

Executing

curl -X POST 'http://<IP>:8089/api/v1/preferences' \
  -H 'accept: application/vnd.api+json' \
  -H 'Authorization: Bearer <TOKEN>' \
  -H 'Content-Type: application/json' -d '{"name": "israeli-bank-importer", "data": "true"}'

Now I passed the first error and get another one

israeli-bank-firefly-importer    | 
israeli-bank-firefly-importer    | > israeli-bank-firefly-importer@0.6.5 start
israeli-bank-firefly-importer    | > node src/index.js
israeli-bank-firefly-importer    | 
israeli-bank-firefly-importer    | [2023-02-13 10:54:49.028] INFO (18): Getting state from firefly...
israeli-bank-firefly-importer    | [2023-02-13 10:54:49.097] INFO (18): Getting scrap data...
israeli-bank-firefly-importer    | [2023-02-13 10:54:49.098] ERROR (18): Fatal error
israeli-bank-firefly-importer    |     err: {
israeli-bank-firefly-importer    |       "type": "TypeError",
israeli-bank-firefly-importer    |       "message": "Cannot read properties of undefined (reading 'hapoalim_<user>')",
israeli-bank-firefly-importer    |       "stack":
israeli-bank-firefly-importer    |           TypeError: Cannot read properties of undefined (reading 'hapoalim_<user>')
israeli-bank-firefly-importer    |               at getLastImport (file:///app/src/importer/last-import-helper.js:35:12)
israeli-bank-firefly-importer    |               at file:///app/src/importer/scrapper.js:52:19
israeli-bank-firefly-importer    |               at Array.map (<anonymous>)
israeli-bank-firefly-importer    |               at getFlatUsers (file:///app/src/importer/scrapper.js:50:6)
israeli-bank-firefly-importer    |               at doImport (file:///app/src/importer/index.js:33:21)
israeli-bank-firefly-importer    |               at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
israeli-bank-firefly-importer    |               at async run (file:///app/src/index.js:12:5)
israeli-bank-firefly-importer    |               at async file:///app/src/index.js:34:3
israeli-bank-firefly-importer    |     }
israeli-bank-firefly-importer exited with code 0

[itairaz1]

Thank you @myakove for reporting about the issue I understand the source of the issue, I will work on a fix

[itairaz1]

A fix was released in 1.2.2 Can you please test it and let me know if you have other troubles? Please note that Hapoalim is not yet tested, so any help will be useful

[myakove]

Thanks

Now I get (from shell, no docker)

 err: {
      "type": "Error",
      "message": "Protocol error (Performance.enable): Target closed.",
      "stack":
          Error: Protocol error (Performance.enable): Target closed.
              at /home/myakove/git/israeli-bank-firefly-importer/node_modules/puppeteer/lib/cjs/puppeteer/common/Connection.js:208:63
              at new Promise (<anonymous>)
              at CDPSession.send (/home/myakove/git/israeli-bank-firefly-importer/node_modules/puppeteer/lib/cjs/puppeteer/common/Connection.js:207:16)
              at Page._initialize (/home/myakove/git/israeli-bank-firefly-importer/node_modules/puppeteer/lib/cjs/puppeteer/common/Page.js:181:26)
              at Page.create (/home/myakove/git/israeli-bank-firefly-importer/node_modules/puppeteer/lib/cjs/puppeteer/common/Page.js:168:20)
              at /home/myakove/git/israeli-bank-firefly-importer/node_modules/puppeteer/lib/cjs/puppeteer/common/Target.js:72:88
              at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
              at async Promise.all (index 0)
              at async BrowserContext.pages (/home/myakove/git/israeli-bank-firefly-importer/node_modules/puppeteer/lib/cjs/puppeteer/common/Browser.js:420:23)
              at async Promise.all (index 0)
    }

and from docker:

israeli-bank-firefly-importer    | 
israeli-bank-firefly-importer    | > israeli-bank-firefly-importer@0.6.5 start
israeli-bank-firefly-importer    | > node src/index.js
israeli-bank-firefly-importer    | 
israeli-bank-firefly-importer    | [2023-02-13 22:11:08.260] INFO (18): Getting state from firefly...
israeli-bank-firefly-importer    | [2023-02-13 22:11:08.317] INFO (18): Getting scrap data...
israeli-bank-firefly-importer    | [2023-02-13 22:11:08.580] ERROR (18): Fatal error
israeli-bank-firefly-importer    |     err: {
israeli-bank-firefly-importer    |       "type": "Error",
israeli-bank-firefly-importer    |       "message": "Failed to launch the browser process!\n[0213/201108.561367:ERROR:zygote_host_impl_linux.cc(100)] Running as root without --no-sandbox is not supported. See https://crbug.com/638180.\n\n\nTROUBLESHOOTING: https://github.com/puppeteer/puppeteer/blob/main/docs/troubleshooting.md\n",
israeli-bank-firefly-importer    |       "stack":
israeli-bank-firefly-importer    |           Error: Failed to launch the browser process!
israeli-bank-firefly-importer    |           [0213/201108.561367:ERROR:zygote_host_impl_linux.cc(100)] Running as root without --no-sandbox is not supported. See https://crbug.com/638180.
israeli-bank-firefly-importer    |           
israeli-bank-firefly-importer    |           
israeli-bank-firefly-importer    |           TROUBLESHOOTING: https://github.com/puppeteer/puppeteer/blob/main/docs/troubleshooting.md
israeli-bank-firefly-importer    |           
israeli-bank-firefly-importer    |               at onClose (/app/node_modules/puppeteer/lib/cjs/puppeteer/node/BrowserRunner.js:193:20)
israeli-bank-firefly-importer    |               at Interface.<anonymous> (/app/node_modules/puppeteer/lib/cjs/puppeteer/node/BrowserRunner.js:183:68)
israeli-bank-firefly-importer    |               at Interface.emit (node:events:524:35)
israeli-bank-firefly-importer    |               at Interface.close (node:internal/readline/interface:534:10)
israeli-bank-firefly-importer    |               at Socket.onend (node:internal/readline/interface:260:10)
israeli-bank-firefly-importer    |               at Socket.emit (node:events:524:35)
israeli-bank-firefly-importer    |               at endReadableNT (node:internal/streams/readable:1359:12)
israeli-bank-firefly-importer    |               at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
israeli-bank-firefly-importer    |     }
israeli-bank-firefly-importer exited with code 0

this error came from puppeteer I think

[myakove]

seems like the issue is in israeli-bank-scrapers

when I run this (index.mjs)

import { CompanyTypes, createScraper } from "israeli-bank-scrapers";

(async function () {
  try {
    // read documentation below for available options
    const options = {
      companyId: CompanyTypes.hapoalim,
      startDate: new Date("2020-05-01"),
      combineInstallments: false,
      showBrowser: true,
    };

    // read documentation below for information about credentials
    const credentials = {
      userCode: "username",
      password: "password",
    };

    const scraper = createScraper(options);
    const scrapeResult = await scraper.scrape(credentials);

    if (scrapeResult.success) {
      scrapeResult.accounts.forEach((account) => {
        console.log(
          `found ${account.txns.length} transactions for account number ${account.accountNumber}`
        );
      });
    } else {
      throw new Error(scrapeResult.errorType);
    }
  } catch (e) {
    console.error(`scraping failed for the following reason: ${e.message}`);
  }
})();

I get: scraping failed for the following reason: Protocol error (Performance.enable): Target closed.

I run it with showBrowser: true, so I see the browser opened but get error Error code: 159 and the page is not loaded.

[myakove]

BTW is this needed?

curl -X POST 'http://<IP>:8089/api/v1/preferences' \
  -H 'accept: application/vnd.api+json' \
  -H 'Authorization: Bearer <TOKEN>' \
  -H 'Content-Type: application/json' -d '{"name": "israeli-bank-importer", "data": "true"}'

If so, maybe you can execute it as part of your code if not exists or at least add it to the README

[itairaz1]

@myakove As you said there is an issue with israeli-bank-scrapers, can you please check with israeli-bank-scrapers developers here: https://github.com/eshaham/israeli-bank-scrapers Which OS are you using? About docker - I'm not sure Docker works, I need to check it and maybe add fixes What you send is not needed, but this is a good workaround to the bug that I had, it should already fixed.

[myakove]

Using Fedora 37 And I need docker to work, I run only on docker in this server.

[myakove]

Dockerfile that work:

FROM node:14-slim

# Install latest chrome dev package and fonts to support major charsets (Chinese, Japanese, Arabic, Hebrew, Thai and a few others)
# Note: this installs the necessary libs to make the bundled version of Chromium that Puppeteer
# installs, work.
RUN apt-get update \
    && apt-get install -y wget gnupg \
    && wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
    && sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' \
    && apt-get update \
    && apt-get install -y libxshmfence1 libglu1 google-chrome-stable fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst fonts-freefont-ttf libxss1 \
    --no-install-recommends \
    && rm -rf /var/lib/apt/lists/*

# If running Docker >= 1.13.0 use docker run's --init arg to reap zombie processes, otherwise
# uncomment the following lines to have `dumb-init` as PID 1
# ADD https://github.com/Yelp/dumb-init/releases/download/v1.2.2/dumb-init_1.2.2_x86_64 /usr/local/bin/dumb-init
# RUN chmod +x /usr/local/bin/dumb-init
# ENTRYPOINT ["dumb-init", "--"]

# Uncomment to skip the chromium download when installing puppeteer. If you do,
# you'll need to launch puppeteer with:
#     browser.launch({executablePath: 'google-chrome-stable'})
# ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD true

# Install puppeteer so it's available in the container.
RUN npm init -y &&  \
    npm i puppeteer \
    # Add user so we don't need --no-sandbox.
    # same layer as npm install to keep re-chowned files from using up several hundred MBs more space
    && groupadd -r pptruser && useradd -r -g pptruser -G audio,video pptruser \
    && mkdir -p /home/pptruser/Downloads \
    && mkdir -p /home/pptruser/app \
    && mkdir -p /home/pptruser/config \
    && chown -R pptruser:pptruser /home/pptruser \
    && chown -R pptruser:pptruser /home/pptruser/app \
    && chown -R pptruser:pptruser /home/pptruser/config \
    && chown -R pptruser:pptruser /node_modules \
    && chown -R pptruser:pptruser /package.json \
    && chown -R pptruser:pptruser /package-lock.json

# Run everything after as non-privileged user.
USER pptruser

WORKDIR /home/pptruser/app
COPY ["package.json", "package-lock.json*", "./"]

RUN npm ci
# the rest of your dockerfile here
COPY . .

ENV IBFI_CONFIG_PATH='/home/pptruser/config'

CMD ["npm", "start"]

[myakove]

One issue left (Bank scrap working great :) )

israeli-bank-firefly-importer    | [2023-02-17 21:13:36.640] ERROR (19): max  failed with type TIMEOUT: waiting for redirect from https://www.max.co.il/homepage/welcome

[itairaz1]

Well done! thank you for the contribution. This max timeout issue is looks like another scrapper issue. If you would like to increase timeout you can do that by setting in the config file:

scraper:
  timeout: 120000

[myakove]

scraper:
  timeout: 120000

Under which section should I put it?

[myakove]

Found it in config/example.yaml

Doesn't seem to help

[myakove]

https://github.com/eshaham/israeli-bank-scrapers/issues/759

[myakove]

Closing this, since my last issue (Max is not working) is not related to this repo.