tusharojha / web_scraper

A very basic web scraper implementation to scrap html elements from a web page.
https://pub.dev/packages/web_scraper
Apache License 2.0
81 stars 41 forks source link

Not work in Flutter Web Application #70

Open ZotInfo opened 2 years ago

ZotInfo commented 2 years ago

I'm developing an investment application for a small group of investors, I need to read the google news, however, when I try to read the page, using flutter web, it returns the error in reading, the same occurs when I use the http plugin .

When I fetch data from an API using http, it works perfectly.

My code

String url = "https://www.google.com", 
page = "/search?q=${search}&hl=pt-BR&gl=BR&ceid=BR:pt-419";

print('getUltimasNoticias: $url$page');
final webScraper = WebScraper(url);
if (await webScraper.loadWebPage(page)) {
  final divElements = webScraper.getElement('div.NiLAwe y6IFtc R7GTQ keNKEd j7vNaf nID9nc', []);
  print("divElements: $divElements");
} else {
  print('Cannot load url');
}

Error and stacktrace

==================================================
Tela: 'NewsHelper'
Função: 'getUltimasNoticias'
Exception: Instance of 'WebScraperException'
StackTrace: C:/b/s/w/ir/cache/builder/src/out/host_debug/dart-sdk/lib/_internal/js_dev_runtime/private/ddc_runtime/errors.dart 251:49  throw_
packages/web_scraper/web_scraper.dart 62:9                                                                                 loadWebPage
C:/b/s/w/ir/cache/builder/src/out/host_debug/dart-sdk/lib/_internal/js_dev_runtime/patch/async_patch.dart 60:31            <fn>
C:/b/s/w/ir/cache/builder/src/out/host_debug/dart-sdk/lib/async/zone.dart 1690:54                                          runBinary
C:/b/s/w/ir/cache/builder/src/out/host_debug/dart-sdk/lib/async/future_impl.dart 174:22                                    handleError
C:/b/s/w/ir/cache/builder/src/out/host_debug/dart-sdk/lib/async/future_impl.dart 778:46                                    handleError
C:/b/s/w/ir/cache/builder/src/out/host_debug/dart-sdk/lib/async/future_impl.dart 799:13                                    _propagateToListeners
C:/b/s/w/ir/cache/builder/src/out/host_debug/dart-sdk/lib/async/future_impl.dart 609:5                                     [_completeError]
C:/b/s/w/ir/cache/builder/src/out/host_debug/dart-sdk/lib/async/future_impl.dart 665:7                                     callback
C:/b/s/w/ir/cache/builder/src/out/host_debug/dart-sdk/lib/async/schedule_microtask.dart 40:11                              _microtaskLoop
C:/b/s/w/ir/cache/builder/src/out/host_debug/dart-sdk/lib/async/schedule_microtask.dart 49:5                               _startMicrotaskLoop
C:/b/s/w/ir/cache/builder/src/out/host_debug/dart-sdk/lib/_internal/js_dev_runtime/patch/async_patch.dart 166:15           <fn>

==================================================
andydean565 commented 2 years ago

have you checked the web console ? was having a simliar issue with cors security

and the following link helped https://stackoverflow.com/questions/65630743/how-to-solve-flutter-web-api-cors-error-only-with-dart-code/66879350#66879350

but does include editing the flutter source code