chromium is opening but not doing anything

[ohadperry]

Macbook Apple M2

what I'm seeing:

I don't think it's related to specific scraper since I've tried leumi also

full code

 const { getPuppeteerConfig } =  require('israeli-bank-scrapers-core');

// local chromium path
//Users/ohadperry/.nvm/versions/node/v20.9.0/lib/node_modules/download-chromium/.local-chromium/chromium-mac-499413/chrome-mac/Chromium.app/Contents/MacOS/Chromium
const chromiumVersion = getPuppeteerConfig().chromiumRevision;
const { CompanyTypes, createScraper } = require('israeli-bank-scrapers-core');

(async function() {
    try {
        // read documentation below for available options
        const options = {
            companyId: CompanyTypes.leumi,
            startDate: new Date('2024-07-07'),
            combineInstallments: false,
            showBrowser: true,
            executablePath: '/Users/ohadperry/.nvm/versions/node/v20.9.0/lib/node_modules/download-chromium/.local-chromium/chromium-mac-499413/chrome-mac/Chromium.app/Contents/MacOS/Chromium'
        };

        // read documentation below for information about credentials
        const credentials = {
            username: process.env.MIZRACHI_USERNAME,
            password: process.env.MIZRACHI_PASSWORD
        };

        const scraper = createScraper(options);
        const scrapeResult = await scraper.scrape(credentials);

        if (scrapeResult.success) {
            scrapeResult.accounts.forEach((account) => {
                console.log(`found ${account.txns.length} transactions for account number ${account.accountNumber}`);
                for (const txn of account.txns) {
                    console.log(`txn is ${JSON.stringify(txn)}`);
                }
            });
        }
        else {
            throw new Error(scrapeResult.errorType);
        }
    } catch(e) {
        console.error(`scraping failed for the following reason: ${e.message}`);
    }
})();

[ohadperry]

plus if there is an example of github actions or aws cron as to how to run preiodically on the could without the localhost dependency it would be great. I can investigate myself with a little help. great project btw!!

[baruchiro]

I suggest waiting for #862 to be merged because it adds a huge upgrade to Pupetteer.

[baruchiro]

Please check with the latest release

[ohadperry]

I did. it still stuck

my package.json

{
  "name": "bankscraper",
  "version": "1.0.0",
  "main": "index.js",
  "author": "ohad@bllink.co",
  "license": "MIT",
  "dependencies": {
    "download-chromium": "^3.0.0",
    "israeli-bank-scrapers-core": "^5.0.1"
  }
}

[ohadperry]

also tried adding "israeli-bank-scrapers": "^5.0.1" same result. maybe it's the chromiumVersion. it changed but download-chromium versionNum doesn't download a new version of it

[ohadperry]

@baruchiro is there a github action yml you know works for this library? I think it will be the simplest solution

[baruchiro]

It sounds like a good idea, I don't remember if I already thought about that.

[ohadperry]

this might work

https://github.com/marketplace/actions/puppeteer-headful

[ohadperry]

tried this and got scraping failed for the following reason: GENERIC

scrapeResult: {"success":false,"errorType":"GENERIC","errorMessage":"Error: failed to find element matching selector \"a[href*=\"/osh/legacy/legacy-Osh-p420\"]\""}

name: Bllink Auto detect failed bank charges
on:
  pull_request:
    branches: [ production  ]
jobs:
  markeFailedBankCharges:
    name: Install Dependencies
    runs-on: ubuntu-latest
    strategy:
      matrix:
        node-version: [ 20.x ]

    steps:
      - uses: actions/checkout@v3
      - name: Install Dependencies
        uses: actions/setup-node@v2
        env:
          PUPPETEER_SKIP_CHROMIUM_DOWNLOAD: "true"
        with:
          args: install

      - name: Install bllink dependencies
        run: yarn install --dev

      - name: Failed Bank Charges
#        uses: mujo-code/puppeteer-headful@16.6.0
#        with:
#          args: yarn run autoBankCharge
        run: yarn autoBankCharge
        env:
          CI: "true"
          MIZRACHI_USERNAME: ${{ secrets.MIZRACHI_USERNAME }}
          MIZRACHI_PASSWORD: ${{ secrets.MIZRACHI_PASSWORD }}

[baruchiro]

The project's tests run headlessly, maybe you can configure them and run the tests and you will get the output.

[ohadperry]

@baruchiro not sure what you mean by that. is it working for other users? I tried running it from a docker and got this error (not finding relevant elements)

https://github.com/daniel-hauser/moneyman/issues/302

[baruchiro]

I'm talking about the Github Actions. It is not my fault you joined two subjects in one issue 😂

[ohadperry]

@baruchiro either way it's not scraping due to this error Error: failed to find element matching selector "a[href*="/osh/legacy/legacy-Osh-p420"]". can you help?

[ohadperry]

I can close this issue and open another one if that helps

[baruchiro]

@baruchiro either way it's not scraping due to this error Error: failed to find element matching selector "a[href*="/osh/legacy/legacy-Osh-p420"]". can you help?

Make sure you're using the latest israeli-bank-scrapers

[ohadperry]

@baruchiro I upgraded to 5.0.1, rebuilt the docker image and re ran it. still the same error. is there anything else I can do to help with the debugging here?

[ohadperry]

I also added a leumi account to check the issue is limited to mizrachi

[ohadperry]

אין לינק לדיסקורד. בכל מקרה, זה לא נפתר כי שידרגתי לגרסה 5.0.1. האם אני צריך להעביר משהו מיוחד בoptions ?

[baruchiro]

OK let's pause.

I need to understand what exactly we are talking about. And yes, if you can open an issue for each problem it will be helpful.

• Chromium is opening but not doing anything?
• Mizrahi is failing?
• Run on Github Actions?

(I deleted the Discord message because I was confused between this project and brafdlog/caspion...)

[ohadperry]

closing this one and opened a new one https://github.com/eshaham/israeli-bank-scrapers/issues/877