ulixee / unblocked

A suite of tools for protecting the web's open knowledge.
MIT License
130 stars 12 forks source link

kasada is not working with #68

Open orithecapper opened 1 year ago

orithecapper commented 1 year ago

image

orithecapper commented 1 year ago

const { Agent } = require('@ulixee/unblocked-agent');

(async () => {
  const agent = new Agent({
    // browserEngine: new Chrome113(),
    options: {
      showChrome: true,
    },
  });
  agent.hook({
    onNewBrowser(browser) {
      browser.engine.launchArguments.push('--no-startup-window', '--disable-background-networking');
    },
  });
  const page = await agent.newPage();
  await page.goto('https://www.canadagoose.com/us/en/trillium-parka-heritage-6660L.html');
  await page.waitForLoad('PaintingStable');

  // const outerHTML = await page.mainFrame.outerHTML();
  // const title = await page.evaluate('document.title');
  // const intro = await page.evaluate(`document.querySelector('p').textContent`);
  // console.log(title)
  // await agent.close();
})();
blakebyrnes commented 1 year ago

Your setup seems to not include any of the plugins that make the browser look like a consumer browser. You might find it easier to use Hero, which automatically included both the default browser emulator and the human emulator.

pashkasan commented 1 year ago

i told u i will come back) lack of examples how to use extensive way, like using agent with default browser emulation and different browsers example provided is nothing, hard to implement,not all super coders) show how to use power of unblocked, that the way and expose cdp and how is catch postmessage events?? and do interception, need custom stuff

blakebyrnes commented 1 year ago

@pashkasan yeah, this library is not well documented because it's kind of an internal system for Hero (https://github.com/ulixee/hero) (https://ulixee.org/docs/hero). It's meant to be a super-coder library if you want to get low level. Ideally, would love to document it so it can be usable too, but it's hard to use without the niceties of Hero, so it's not currently a high priority.

FWIW, this is separated into its own repository from Hero so that it can follow the entire lifecycle of cat/mouse evasion/detection.

pashkasan commented 1 year ago

i can send u private codes of puppetter, how to catch not popup but css popups many sites use it now and lock screen

blakebyrnes commented 1 year ago

i can send u private codes of puppetter, how to catch not popup but css popups many sites use it now and lock screen

I'm confused. Are you adding this comment on the right issue? We have a feature in Hero called Flow handlers that are meant to deal with css popups that interrupt the intended flow of your script (https://ulixee.org/docs/hero/basic-client/flow-handling)

pashkasan commented 1 year ago

for sure u need more usabale example how to use power of your project to emulate real user, ua and etc this will grow community and codefor sure more coders MORE POWER

blakebyrnes commented 1 year ago

for sure u need more usabale example how to use power of your project to emulate real user, ua and etc this will grow community and codefor sure more coders MORE POWER

Ah, yes. Would be very helpful. Are you using low level Agent? Not Hero? There's a little more documentation on Hero (although we could certainly use more)

pashkasan commented 1 year ago

Flow handlers, i read this article, but not find solution to my question how to handle event in nodejs ie window.addEventListener('message', e => {console.log('ev', e);});

pashkasan commented 1 year ago

for sure u need more usabale example how to use power of your project to emulate real user, ua and etc this will grow community and codefor sure more coders MORE POWER

Ah, yes. Would be very helpful. Are you using low level Agent? Not Hero? There's a little more documentation on Hero (although we could certainly use more)

yes, low level for now, tried you project several times, no luck, but looking to use it with agent, not hero i think it overloaded with command center

pashkasan commented 1 year ago

what i need = low cpu profile, raw cdp for some moments, browser emulator u implemented with great mitm, chromes never low profile, i talking aout communications betwwen soft and dev proto, eat many cpu on multiple threads not talking aout load balancing, but can share my practical implementations

blakebyrnes commented 1 year ago

Flow handlers, i read this article, but not find solution to my question how to handle event in nodejs ie window.addEventListener('message', e => {console.log('ev', e);});

Flow Handlers were an answer to your comment about css popups. If you want to run raw javascript on the page, you would just use page.evaluate("window.addEventListener('message', e => {console.log('ev', e);});")

blakebyrnes commented 1 year ago

what i need = low cpu profile, raw cdp for some moments, browser emulator u implemented with great mitm

If you'd like help, maybe the best option is to connect on Discord. I understand your desire to have better documentation on Agent. I'm not sure Agent is going to have a smooth user experience in the near-term though, as almost all of our dom handling is on the Hero side.

pashkasan commented 1 year ago

yes, but need to catch in nodejs soft pptr

    const listenForPageEvent = function (page, type) {
        log('listenForPageEvent init for ', page.url());
        return page.evaluate(type => {
            window.addEventListener(type, e => {

                window.onMessageReceivedEvent({
                    type,
                    event: e.data,
                    origin: e.origin
                });
            });
        }, type);
    };

    await page.exposeFunction('onMessageReceivedEvent', e => {

        let event = e.event;

        switch (event) {

    got it
pashkasan commented 1 year ago

i will do

blakebyrnes commented 1 year ago

yes, but need to catch in nodejs soft pptr

    const listenForPageEvent = function (page, type) {
        log('listenForPageEvent init for ', page.url());
        return page.evaluate(type => {
            window.addEventListener(type, e => {

                window.onMessageReceivedEvent({
                    type,
                    event: e.data,
                    origin: e.origin
                });
            });
        }, type);
    };

    await page.exposeFunction('onMessageReceivedEvent', e => {

        let event = e.event;

        switch (event) {

  got it

Let's discuss this here: https://github.com/ulixee/unblocked/discussions/69

orithecapper commented 1 year ago

Your setup seems to not include any of the plugins that make the browser look like a consumer browser. You might find it easier to use Hero, which automatically included both the default browser emulator and the human emulator.

Hey, both of them didnt work...

for ref, I tried using my computer@work , and it did work

but when for example trying to go to nike.com and "add cart" they do some how detect that its a virtualised thing

kasada is really impresive I think they have some interesting way to detect fingerprints

orithecapper commented 1 year ago

Your setup seems to not include any of the plugins that make the browser look like a consumer browser. You might find it easier to use Hero, which automatically included both the default browser emulator and the human emulator.

Hero fails to emulate , also nike.com "Add cart" is failing hard too not that I care much, its just a point that ulixee is being detected somehow imo you can make a lot of money if you can figure out what they do , and sell your own bot protection and not have many attacks know...

anyways, about kasada, they say that they want to make "Attacking too costly" and "waste attacker resources" in their blogs.. you may be interested