huds2 / gtrends34

Quick hack for getting Google Trends info using selenium
0 stars 0 forks source link

Could not find central directory end #2

Closed makorne closed 5 months ago

makorne commented 5 months ago

Hi! Thank you for your interesting crate. But how to use it?

ChromeDriver does not exist! Fetching...
Getting installed Chrome version...
Currently installed Chrome version: 124
thread 'main' panicked at /.cargo/git/checkouts/rust-undetected-chromedriver-791f14f9b72420ec/78e38e4/src/lib.rs:28:43:
called `Result::unwrap()` on an `Err` value: InvalidArchive("Could not find central directory end")
huds2 commented 5 months ago

Hello! This isn't really a complete project, but a quick hack. The problem you were facing seemed to be related to Google changing download URLs for the chromedriver. I've updated my fork of the Rust-undetected-chromedriver crate to use the newer URL and it seems to be working now.

You can use the crate the following way:

let trends = gtrends34::GTrends::new("/tmp/directory/to/store/the/downloaded/files/").await?;
let result = trends.get_keyword("keyword").await?;
makorne commented 5 months ago

Thanks for prompt reply!

But now the new error:

Starting chromedriver...
Starting ChromeDriver 124.0.6367.207 (a9001a6e39fbaa559510ca866052950457dd4e6b-refs/branch-heads/6367_201@{#3}) on port 3813
Only local connections are allowed.
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
ChromeDriver was started successfully.
error creating new session: webdriver did not create session: session not created: DevToolsActivePort file doesn't exist
thread 'main' panicked at /home/.cargo/git/checkouts/rust-undetected-chromedriver-791f14f9b72420ec/1e97d6e/src/lib.rs:135:25:
called `Option::unwrap()` on a `None` value

And if I add a googled option caps.add_chrome_arg("--remote-debugging-port=9222")?;

ChromeDriver was started successfully.
thread 'tokio-runtime-worker' panicked at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fantoccini-0.19.3/src/session.rs:921:34:
internal error: entered unreachable code: received unknown error (chrome not reachable) for INTERNAL_SERVER_ERROR status code
stack backtrace:
   0: rust_begin_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:645:5
   1: core::panicking::panic_fmt
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/panicking.rs:72:14
   2: fantoccini::session::Session<C>::issue_wd_cmd::{{closure}}
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fantoccini-0.19.3/src/session.rs:921:34
   3: <T as futures_util::fns::FnOnce1<A>>::call_once
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/fns.rs:15:9
   4: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/future/map.rs:57:73
   5: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/lib.rs:91:13
   6: <futures_util::future::either::Either<A,B> as core::future::future::Future>::poll
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/either.rs:109:32
   7: fantoccini::session::Ongoing::poll
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fantoccini-0.19.3/src/session.rs:383:51
   8: <fantoccini::session::Session<C> as core::future::future::Future>::poll
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fantoccini-0.19.3/src/session.rs:448:30
   9: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:328:17
  10: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/loom/std/unsafe_cell.rs:16:9
  11: tokio::runtime::task::core::Core<T,S>::poll
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:317:13
  12: tokio::runtime::task::harness::poll_future::{{closure}}
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:485:19
  13: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/panic/unwind_safe.rs:272:9
  14: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  15: __rust_try
  16: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  17: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  18: tokio::runtime::task::harness::poll_future
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:473:18
  19: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:208:27
  20: tokio::runtime::task::harness::Harness<T,S>::poll
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:153:15
  21: tokio::runtime::task::raw::poll
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:271:5
  22: tokio::runtime::task::raw::RawTask::poll
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:201:18
  23: tokio::runtime::task::LocalNotified<S>::run
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/mod.rs:427:9
  24: tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:639:17
  25: tokio::runtime::coop::with_budget
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/coop.rs:107:5
  26: tokio::runtime::coop::budget
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/coop.rs:73:5
  27: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:575:9
  28: tokio::runtime::scheduler::multi_thread::worker::Context::run
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:526:24
  29: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:491:21
  30: tokio::runtime::context::scoped::Scoped<T>::set
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context/scoped.rs:40:9
  31: tokio::runtime::context::set_scheduler::{{closure}}
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context.rs:176:26
  32: std::thread::local::LocalKey<T>::try_with
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/thread/local.rs:284:16
  33: std::thread::local::LocalKey<T>::with
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/thread/local.rs:260:9
  34: tokio::runtime::context::set_scheduler
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context.rs:176:9
  35: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:486:9
  36: tokio::runtime::context::runtime::enter_runtime
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context/runtime.rs:65:16
  37: tokio::runtime::scheduler::multi_thread::worker::run
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:478:5
  38: tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:447:45
  39: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/task.rs:42:21
  40: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:328:17
  41: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/loom/std/unsafe_cell.rs:16:9
  42: tokio::runtime::task::core::Core<T,S>::poll
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:317:13
  43: tokio::runtime::task::harness::poll_future::{{closure}}
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:485:19
  44: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/panic/unwind_safe.rs:272:9
  45: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  46: __rust_try
  47: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  48: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  49: tokio::runtime::task::harness::poll_future
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:473:18
  50: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:208:27
  51: tokio::runtime::task::harness::Harness<T,S>::poll
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:153:15
  52: tokio::runtime::task::raw::poll
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:271:5
  53: tokio::runtime::task::raw::RawTask::poll
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:201:18
  54: tokio::runtime::task::UnownedTask<S>::run
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/mod.rs:464:9
  55: tokio::runtime::blocking::pool::Task::run
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:159:9
  56: tokio::runtime::blocking::pool::Inner::run
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:513:17
  57: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
             at /home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:471:13
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
error creating new session: webdriver server disconnected: WebDriver session was closed while waiting
thread 'main' panicked at /home/.cargo/git/checkouts/rust-undetected-chromedriver-791f14f9b72420ec/1e97d6e/src/lib.rs:135:25:
called `Option::unwrap()` on a `None` value

The caps.add_chrome_arg("--remote-debugging-pipe")?;// https://bugs.chromium.org/p/chromedriver/issues/detail?id=4403#c35 Does not help too. Any ideas?

huds2 commented 5 months ago

I was unable to recreate the error. After a bit of googling I've found a StackOverflow entry that suggests using --disable-dev-shm-usage argument.

Also this entry talks about Chrome crashing when running as root.

Not sure if any of this will help, but I can't test the solutions on my end.

makorne commented 5 months ago

I already tried using --disable-dev-shm-usage argument. and many others from google :( My chrome not crashing but dont have debugging connect :( What OS do you use?

huds2 commented 5 months ago

I'm using Debian Bookworm. You can try updating your Chrome install. I'm currently using chromium and this is the output of chromium --version on my machine:

Chromium 125.0.6422.60 built on Debian 12.5, running on Debian 12.5
makorne commented 5 months ago

I have Chromium too: Version 119.0.6045.199 (Official Build) Fedora Project (64-bit) 37 But rust-undetected-chromedriver download and patch only chrome from google, is`nt it? How do you use chromium with rust-undetected-chromedriver ?

huds2 commented 5 months ago

The Rust-undetected-chromedriver crate checks the installed chrome/chromium version, downloads the chromedriver that matches the version and then patches it to avoid detection by websites.

makorne commented 5 months ago

I have both chrome/chromium... May be this is a problem ...

makorne commented 5 months ago

Remove chrome, chromedriver and reinstall chromium does not help :(

Starting chromedriver...
Starting ChromeDriver 119.0.6045.105 (38c72552c5e15ba9b3117c0967a0fd105072d7c6-refs/branch-heads/6045@{#1103}) on port 2155
Only local connections are allowed.
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
ChromeDriver was started successfully.
error creating new session: webdriver did not create session: session not created: DevToolsActivePort file doesn't exist
thread 'main' panicked at /home/.cargo/git/checkouts/rust-undetected-chromedriver-791f14f9b72420ec/1e97d6e/src/lib.rs:135:25:
called `Option::unwrap()` on a `None` value
makorne commented 5 months ago

On Debian 12.5 it works! Thank you!!

huds2 commented 5 months ago

That's interesting. Probably has something to do with the chromium version in the Debian and Fedora repositories.