Important!
The scrapers are set to use timezone
Asia/Jerusalem
to avoid conflicts in case you're running the scrapers outside Israel.
What you can find here is scrapers for all major Israeli banks and credit card companies. That's the plan at least. Currently only the following banks are supported:
To use this you will need to have Node.js >= 16.x installed.
To use these scrapers you'll need to install the package from npm:
npm install israeli-bank-scrapers --save
Then you can simply import and use it in your node module:
import { CompanyTypes, createScraper } from 'israeli-bank-scrapers';
(async function() {
try {
// read documentation below for available options
const options = {
companyId: CompanyTypes.leumi,
startDate: new Date('2020-05-01'),
combineInstallments: false,
showBrowser: true
};
// read documentation below for information about credentials
const credentials = {
username: 'vr29485',
password: 'sometingsomething'
};
const scraper = createScraper(options);
const scrapeResult = await scraper.scrape(credentials);
if (scrapeResult.success) {
scrapeResult.accounts.forEach((account) => {
console.log(`found ${account.txns.length} transactions for account number ${account.accountNumber}`);
});
}
else {
throw new Error(scrapeResult.errorType);
}
} catch(e) {
console.error(`scraping failed for the following reason: ${e.message}`);
}
})();
Check the options declaration here for available options.
Regarding credentials, you should provide the relevant credentials for the chosen company. See this file with list of credentials per company.
The structure of the result object is as follows:
{
success: boolean,
accounts: [{
accountNumber: string,
balance?: number, // Account balance. Not implemented for all accounts.
txns: [{
type: string, // can be either 'normal' or 'installments'
identifier: int, // only if exists
date: string, // ISO date string
processedDate: string, // ISO date string
originalAmount: double,
originalCurrency: string,
chargedAmount: double,
description: string,
memo: string, // can be null or empty
installments: { // only if exists
number: int, // the current installment number
total: int, // the total number of installments
},
status: string //can either be 'completed' or 'pending'
}],
}],
errorType: "INVALID_PASSWORD"|"CHANGE_PASSWORD"|"ACCOUNT_BLOCKED"|"UNKNOWN_ERROR"|"TIMEOUT"|"GENERIC", // only on success=false
errorMessage: string, // only on success=false
}
You can also use the SCRAPERS
list to get scraper metadata:
import { SCRAPERS } from 'israeli-bank-scrapers';
The return value is a list of scraper metadata:
{
<companyId>: {
name: string, // the name of the scraper
loginFields: [ // a list of login field required by this scraper
'<some field>' // the name of the field
]
}
}
Some companies require two-factor authentication, and as such the scraper cannot be fully automated. When using the relevant scrapers, you have two options:
import { CompanyTypes, createScraper } from 'israeli-bank-scrapers';
import { prompt } from 'enquirer';
// Option 1 - Provide a callback
const result = await scraper.login({
email: relevantAccount.credentials.email,
password: relevantAccount.credentials.password,
phoneNumber,
otpCodeRetriever: async () => {
let otpCode;
while (!otpCode) {
otpCode = await questions('OTP Code?');
}
return otpCode[0];
}
});
// Option 2 - Retrieve a long term otp token (OneZero)
await scraper.triggerTwoFactorAuth(phoneNumber);
// OTP is sent, retrieve it somehow
const otpCode='...';
const result = scraper.getLongTermTwoFactorToken(otpCode);
/*
result = {
success: true;
longTermTwoFactorAuthToken: 'eyJraWQiOiJiNzU3OGM5Yy0wM2YyLTRkMzktYjBm...';
}
*/
This library is deployed automatically to NPM with any change merged into the master branch.
Israeli-bank-scrapers-core
libraryTL;DR this is the same library as the default library. The only difference is that it is using
puppeteer-core
instead ofpuppeteer
which is useful if you are using frameworks like Electron to pack your application.In most cases you will probably want to use the default library (read Getting Started section).
Israeli bank scrapers library is published twice:
israeli-bank-scrapers-core - extremely useful for applications that bundle node_modules
like Electron applications.
The default variation israeli-bank-scrapers is using puppeteer which handles the installation of local chroumium on its' own. This behavior is very handy since it takes care on all the hard work figuring which chromium to download and manage the actual download process. As a side effect it increases node_modules by several hundred megabytes.
The core variation israeli-bank-scrapers-core is using puppeteer-core which is exactly the same library as puppeteer
except that it doesn't download chromium when installed by npm. It is up to you to make sure the specific version of chromium is installed locally and provide a path to that version. It is useful in Electron applications since it doesn't bloat the size of the application and you can provide a much friendlier experience like loading the application and download it later when needed.
To install israeli-bank-scrapers-core
:
npm install israeli-bank-scrapers-core --save
When using the israeli-bank-scrapers-core
it is up to you to make sure the relevant chromium version exists. You must:
puppeteer-core
library being used.israeli-bank-scrapers-core
scrapers.Please read the following to learn more about the process:
import { getPuppeteerConfig } from 'israeli-bank-scrapers-core';
const chromiumVersion = getPuppeteerConfig().chromiumRevision;
2. Once you have the chromium revision, you can either download it manually or use other liraries like [download-chromium](https://www.npmjs.com/package/download-chromium) to fetch that version. The mentioned library is very handy as it caches the download and provide useful helpers like download progress information.
3. provide the path to chromium to the library using the option key `executablePath`.
# Specific definitions per scraper
## Bank Hapoalim scraper
This scraper expects the following credentials object:
```node
const credentials = {
userCode: <user identification code>,
password: <user password>
};
This scraper supports fetching transaction from up to one year.
This scraper expects the following credentials object:
const credentials = {
username: <user name>,
password: <user password>
};
This scraper supports fetching transaction from up to one year.
This scraper expects the following credentials object:
const credentials = {
id: <user identification number>,
password: <user password>,
num: <user identificaiton code>
};
This scraper supports fetching transaction from up to one year (minus 1 day).
This scraper expects the following credentials object:
const credentials = {
id: <user identification number>,
password: <user password>,
num: <user identificaiton code>
};
This scraper supports fetching transaction from up to one year (minus 1 day).
This scraper expects the following credentials object:
const credentials = {
username: <user identification number>,
password: <user password>
};
This scraper supports fetching transaction from up to one year.
These scrapers are essentially identical and expect the following credentials object:
const credentials = {
username: <user name>,
password: <user password>
};
This scraper expects the following credentials object:
const credentials = {
username: <user name>,
password: <user password>
};
This scraper supports fetching transaction from up to one year.
This scraper expects the following credentials object:
const credentials = {
username: <user name>,
password: <user password>
};
This scraper supports fetching transaction from up to one year.
This scraper expects the following credentials object:
const credentials = {
username: <user name>,
password: <user password>
};
This scraper supports fetching transaction from up to one year.
This scraper expects the following credentials object:
const credentials = {
id: <user identification number>,
card6Digits: <6 last digits of card>
password: <user password>
};
This scraper supports fetching transaction from up to one year.
This scraper expects the following credentials object:
const credentials = {
username: <user identification number>,
card6Digits: <6 last digits of card>
password: <user password>
};
This scraper supports fetching transaction from up to one year.
This scraper expects the following credentials object:
const credentials = {
username: <user name>,
password: <user password>,
nationalID: <user national ID>
};
This scraper supports fetching transaction from up to six months.
This scraper expects the following credentials object::
const credentials = {
id: <user identification number>,
password: <user password>
};
These are the projects known to be using this module:
Built something interesting you want to share here? Let me know.
The MIT License