search

skrapeit / skrape.it

A Kotlin-based testing/scraping/parsing library providing the ability to analyze and extract data from HTML (server & client-side rendered). It places particular emphasis on ease of use and a high level of readability by providing an intuitive DSL. It aims to be a testing lib, but can also be used to scrape websites in a convenient fashion.
https://docs.skrape.it
MIT License
808 stars 59 forks source link

[BUG] Fatal Exception when using BrowserFetcher #129

Closed bryandonmarc closed 3 years ago

bryandonmarc commented 3 years ago

Describe the bug Using the build it.skrape:skrapeit-core:1.0.0-alpha8, I get a FATAL EXCEPTION for java.lang.NoSuchFieldError when using BrowserFetcher

Here is the stack trace:

E/AndroidRuntime: FATAL EXCEPTION: main
    Process: com.example.skraper, PID: 8340
    java.lang.NoSuchFieldError: No static field INSTANCE of type Lorg/apache/http/conn/ssl/AllowAllHostnameVerifier; in class Lorg/apache/http/conn/ssl/AllowAllHostnameVerifier; or its superclasses (declaration of 'org.apache.http.conn.ssl.AllowAllHostnameVerifier' appears in /system/framework/framework.jar!classes4.dex)
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.<clinit>(SSLConnectionSocketFactory.java:151)
        at com.gargoylesoftware.htmlunit.httpclient.HtmlUnitSSLConnectionSocketFactory.buildSSLSocketFactory(HtmlUnitSSLConnectionSocketFactory.java:89)
        at com.gargoylesoftware.htmlunit.HttpWebConnection.configureHttpsScheme(HttpWebConnection.java:663)
        at com.gargoylesoftware.htmlunit.HttpWebConnection.createHttpClientBuilder(HttpWebConnection.java:580)
        at com.gargoylesoftware.htmlunit.HttpWebConnection.getHttpClientBuilder(HttpWebConnection.java:541)
        at com.gargoylesoftware.htmlunit.HttpWebConnection.getResponse(HttpWebConnection.java:172)
        at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseFromWebConnection(WebClient.java:1536)
        at com.gargoylesoftware.htmlunit.WebClient.loadWebResponse(WebClient.java:1459)
        at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:447)
        at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:368)
        at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:520)
        at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:501)
        at it.skrape.core.fetcher.BrowserFetcher.fetch(BrowserFetcher.kt:21)
        at it.skrape.core.fetcher.BrowserFetcher.fetch(BrowserFetcher.kt:12)
        at it.skrape.core.Scraper.scrape(Scraper.kt:15)
        at it.skrape.DslKt.extract(Dsl.kt:29)
        at com.example.skraper.MainActivity$onCreate$test$1.invoke(MainActivity.kt:25)
        at com.example.skraper.MainActivity$onCreate$test$1.invoke(MainActivity.kt:11)
        at it.skrape.DslKt.skrape(Dsl.kt:15)
        at com.example.skraper.MainActivity.onCreate(MainActivity.kt:23)
        at android.app.Activity.performCreate(Activity.java:7984)
        at android.app.Activity.performCreate(Activity.java:7973)
        at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1309)
        at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:3329)
        at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:3500)
        at android.app.servertransaction.LaunchActivityItem.execute(LaunchActivityItem.java:83)
        at android.app.servertransaction.TransactionExecutor.executeCallbacks(TransactionExecutor.java:135)
        at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:95)
        at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2049)
        at android.os.Handler.dispatchMessage(Handler.java:106)
        at android.os.Looper.loop(Looper.java:223)
        at android.app.ActivityThread.main(ActivityThread.java:7523)
        at java.lang.reflect.Method.invoke(Native Method)
        at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:592)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:941)

Code Sample Just a basic, bare-bones example of me trying to test out this library

import it.skrape.core.fetcher.BrowserFetcher
import it.skrape.core.htmlDocument
import it.skrape.extract
import it.skrape.skrape

val test = skrape(BrowserFetcher) {
    request { url = "https://web-scraping-playground-site.firebaseapp.com" }
    extract {
        htmlDocument {
            this@skrape
        }
    }
}

println(test)

Expected behavior It should work without errors.

Additional context Gradle configuration:

android {
    compileSdkVersion 30
    buildToolsVersion "30.0.3"

    defaultConfig {
        applicationId "com.example.skraper"
        minSdkVersion 21
        targetSdkVersion 30
        /* ... */
    }    
    compileOptions {
        sourceCompatibility JavaVersion.VERSION_1_8
        targetCompatibility JavaVersion.VERSION_1_8
    }
    kotlinOptions {
        jvmTarget = '1.8'
    }
    /* ... */
}

run via Pixel_3a_API_30 Android 11.0 (Google APIs) x86_64 AVD from Android Studio

christian-draeger commented 3 years ago

duplicate of #127