sokomishalov / skraper

Kotlin/Java library and cli tool for scraping posts and media from various sources with neither authorization nor full page rendering (Facebook, Instagram, Twitter, Youtube, Tiktok, Telegram, Twitch, Reddit, 9GAG, Pinterest, Flickr, Tumblr, Coub, Vimeo, IFunny, VK, Odnoklassniki, Pikabu)
Apache License 2.0
253 stars 42 forks source link

how to use this library in java? and thow use proxy #457

Closed hookover closed 9 months ago

hookover commented 10 months ago

Thank you.

error

Exception in thread "main" java.lang.NoClassDefFoundError: io/ktor/client/HttpClientJvmKt
    at ru.sokomishalov.skraper.client.ktor.KtorSkraperClient.<clinit>(KtorSkraperClient.kt:86)
    at org.example.Main.main(Main.java:11)
Caused by: java.lang.ClassNotFoundException: io.ktor.client.HttpClientJvmKt
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
    ... 2 more

Process finished with exit code 1

main.java

package org.example;

import com.google.gson.Gson;
import kotlinx.coroutines.flow.Flow;
import ru.sokomishalov.skraper.client.ktor.KtorSkraperClient;
import ru.sokomishalov.skraper.model.Post;
import ru.sokomishalov.skraper.provider.facebook.FacebookSkraper;

public class Main {
    public static void main(String[] args) {
        KtorSkraperClient httpClient = new KtorSkraperClient();
        FacebookSkraper skraper = new FacebookSkraper(httpClient);
        Flow<Post> posts=  skraper.getPosts("memes");

        System.out.println( new Gson().toJson(posts));
    }
}

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>spider-java-test</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <maven.compiler.source>21</maven.compiler.source>
        <maven.compiler.target>21</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>ru.sokomishalov.skraper</groupId>
            <artifactId>skrapers</artifactId>
            <version>0.12.1</version>
        </dependency>

        <dependency>
            <groupId>com.google.code.gson</groupId>
            <artifactId>gson</artifactId>
            <version>2.10.1</version>
        </dependency>
    </dependencies>
</project>
sokomishalov commented 9 months ago

Please take a look at this section, all details regarding java interop are there.

Regarding proxy - it can be set up within underlying client transport. (i.e. ktor)