question: how to import/export all cookies from browser?

[sergerdn]

I'm aware that Selenium only exports cookies for the current website. In the past, I've used workarounds like patching Chrome driver to get cookies for all websites or installing a Chrome extension to export/import all cookies. I also know that another way to do this is to parse cookies from the file in the browser profile directly, but I haven't tested it yet.

In the context of using tools like your library or BAS, exporting and importing cookies is a crucial feature.

If you decide to implement exporting and importing cookies with this library or any other tool, it's important to consider the BAS format for exporting cookies to ensure cross-compatibility between this library and BAS. Using a standardized format like JSON for exporting and importing cookies can also help ensure compatibility between different tools and libraries.

// Import dependencies:
require('chromedriver');
const {plugin} = require('selenium-with-fingerprints');
const fs = require('fs');
const {promisify} = require('util');
const writeFileAsync = promisify(fs.writeFile);

async function navigateAndSaveCookies(driver,url,filename) {
    // Navigate to the website:
    await driver.get(url);

    // Wait for the page to load:
    await driver.wait(async () => {
        const pageState = await driver.executeScript('return document.readyState');
        return pageState === 'complete';
    }, 5000);

    // Read cookies and write to file. 
   //  =======>>>>>>>>> Cookies are saved only for the currently opened website. <<<<<<<<===== 
    const cookies = await driver.manage().getCookies();
    await writeFileAsync(filename, JSON.stringify(cookies, null, 2));
}

// Main function:
(async () => {

    // Get a fingerprint from the server:
    const fingerprint = await plugin.fetch('', {
        tags: ['Microsoft Windows', 'Chrome'],
    });

    // Apply fingerprint:
    plugin.useFingerprint(fingerprint);

    // Launch the browser instance:
    const driver = await plugin.launch();

    // Maximize the window:
    await driver.manage().window().maximize();

    await navigateAndSaveCookies(driver,'https://www.google.com/webhp?hl=en','cookies_google.json')
    await navigateAndSaveCookies(driver,'https://yandex.com/','cookies_yandex.json')

    // Quit the browser:
    await driver.quit();
})();

[CheshireCaat]

You can try using CDP to get all cookies, for example with Storage.getCookies or Network.getCookies methods. You can send DevTools commands using this or this method.

Exporting and importing cookies is not the responsibility of plugins, as well as implementing the logic for their compatibility with BAS. I think users should do it on their side.

[sergerdn]

I have successfully met my requirements and resolved the issue. I would like to suggest adding this code to the following directory: https://github.com/CheshireCaat/selenium-with-fingerprints/tree/master/examples, as I believe it could be helpful to others with similar needs and to prevent future redundant questions.

If you are interested, I would be happy to create a pull request for the changes. Alternatively, you can add these changes manually if you prefer.

Please let me know your thoughts.

require('chromedriver'); // do not remove this line

const fs = require('fs');
const os = require('os');
const path = require('path');
const {Builder} = require('selenium-webdriver');
const {Options} = require('selenium-webdriver/chrome');
const {plugin} = require('selenium-with-fingerprints');
const CDP = require('chrome-remote-interface'); // npm install chrome-remote-interface

/**
 * Asynchronously navigates to a given URL in the specified Selenium driver instance.
 * @param {webdriver.WebDriver} driver - The Selenium driver instance.
 * @param {string} url - The URL to navigate to.
 * @param {number} timeout - The maximum amount of time to wait for the page to load, in milliseconds.
 */
async function navigateUrl(driver, url, timeout = 5000) {
    // Navigate to the website:
    await driver.get(url);

    // Wait for the page to load:
    await driver.wait(async () => {
        const pageState = await driver.executeScript('return document.readyState');
        return pageState === 'complete';
    }, timeout);
}

/**
 * Asynchronously retrieves all cookies from a Chrome instance running at the specified debugger address.
 * @param {int} port - The debugger address of the Chrome instance.
 */
async function getAllCookies(port) {
    // Get all cookies using CDP:
    const clientCDP = await CDP({port});
    const {Storage} = clientCDP;
    const {cookies} = await Storage.getCookies();
    console.log('All cookies:', JSON.stringify(cookies, null, 2));
    await clientCDP.close();
}

/**
 * Asynchronously removes the temporary profile directory and its contents.
 * @param {string} profileDir - The path to the temporary profile directory.
 */
async function cleanUpProfileDir(profileDir) {
    while (true) {
        // Wait for the browser to completely close before attempting to delete the profile directory:
        await new Promise(resolve => setTimeout(resolve, 1000));
        try {
            fs.rmSync(profileDir, {recursive: true});
            break;
        } catch (err) {
        }
    }
}

(async () => {
    // Get a fingerprint from the server:
    const fingerprint = await plugin.fetch('', {
        tags: ['Microsoft Windows', 'Chrome'],
    });

    // Apply the fingerprint to the plugin:
    plugin.useFingerprint(fingerprint);

    // Create a temporary directory for the Chrome profile:
    const profileDir = fs.mkdtempSync(path.join(os.tmpdir(), 'chrome-'));

    // Set the Chrome options to use the new profile:
    const chromeOptions = new Options()

    // Spawn a new Chrome instance using the plugin and the temporary profile directory:
    const chrome = await plugin.spawn({headless: false, userDataDir: profileDir});
    // Connect to the Chrome instance using the debugger address:
    chromeOptions.debuggerAddress(`localhost:${chrome.port}`);

    // Launch the browser instance:
    const driver = await new Builder()
        .forBrowser('chrome')
        .setChromeOptions(chromeOptions)
        .build();

    // Maximize the window:
    await driver.manage().window().maximize();

    // Navigate to some websites:
    await navigateUrl(driver, 'https://www.google.com/webhp?hl=en');
    await navigateUrl(driver, 'https://yandex.com/');

    // Get all cookies using CDP:
    await getAllCookies(chrome.port);

    // Quit the driver:
    await driver.quit();
    // Quit the browser:
    await chrome.close();

    // Clean up the temporary profile directory:
    await cleanUpProfileDir(profileDir);

})();

[CheshireCaat]

I recommend not using spawn, given that the plugin has a ready-made method for launching the browser. It's more suitable for cases where you need to migrate code, or for other frameworks without plugins - plus, you will need to manually call the chrome.configure() method after opening each page.

There are ready-made methods in selenium for working with CDP, I gave you the links above. Your cookie retrieval method could be rewritten like this:

async function getAllCookies(driver) {
  const { cookies } = await driver.sendAndGetDevToolsCommand('Storage.getCookies');
  console.log('All cookies:', JSON.stringify(cookies, null, 2));
  return cookies;
}

Or even shorter:

async function getAllCookies(driver) {
  return await driver.sendAndGetDevToolsCommand('Storage.getCookies').then(({ cookies }) => cookies);
}

If you need to add custom settings for the launch like debugger address etc - you can pass the Builer instance as a parameter to the launch method (without calling the build method), check an example here.

[CheshireCaat]

Regarding the example of saving cookies - a bit later i'll add for all frameworks, thanks.

[sergerdn]

Thank you for your suggestion to improve my code. I appreciate it since I mentioned in another issue that I am not a JavaScript developer, and comments from a JavaScript developer can help me improve my experience with JavaScript.

Could you please confirm if you will close this issue once the modifications have been made?

[CheshireCaat]

Yes, of course, i'll mention this ticket in the commit.

[sergerdn]

Hi, @CheshireCaat. You have made a small spelling error. You wrote фn instead of an:

"The cookies.js file - фn example of interaction with all browser cookies."

https://github.com/CheshireCaat/selenium-with-fingerprints/commit/da7dc71cc3ece9b0f270c982fe825f0098100add#diff-49aaa2819e35a856818ecec8c9fa7e1c79ad028d3f44bd749736353cfb51bac9R5

[CheshireCaat]

Didn't notice it right away, thanks.