search

fanyong920 / jvppeteer

Headless Chrome For Java （Java 爬虫）
Apache License 2.0
722 stars 158 forks source link

Cannot construct instance of `com.ruiyun.jvppeteer.entities.Timestamp` #159

Open coolguitar opened 1 day ago

coolguitar commented 1 day ago

com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot construct instance of com.ruiyun.jvppeteer.entities.Timestamp (although at least one Creator exists): no double/Double-argument constructor/factory method to deserialize from Number value (1.731399248104643E12) at [Source: UNKNOWN; line: -1, column: -1] (through reference chain: com.ruiyun.jvppeteer.events.ExceptionThrownEvent["timestamp"]) at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:63) at com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1343) at com.fasterxml.jackson.databind.DeserializationContext.handleMissingInstantiator(DeserializationContext.java:1032) at com.fasterxml.jackson.databind.deser.ValueInstantiator.createFromDouble(ValueInstantiator.java:274) at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromDouble(StdValueInstantiator.java:378) at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromDouble(BeanDeserializerBase.java:1396) at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:175) at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:161) at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129) at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:288) at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:151) at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3984) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2276) at com.fasterxml.jackson.databind.ObjectMapper.treeToValue(ObjectMapper.java:2758) at com.ruiyun.jvppeteer.transport.CDPSession.lambda$onMessage$0(CDPSession.java:134) at com.ruiyun.jvppeteer.transport.Connection.lambda$startEmitEventThread$6(Connection.java:210) at java.lang.Thread.run(Thread.java:748) [WebSocketConnectReadThread-13] INFO com.ruiyun.jvppeteer.transport.WebSocketTransport - Connection closed by us Code: 1000 Reason: Normal closure

fanyong920 commented 1 day ago

给个测试样例

 

樊勇 @.***

 

------------------ 原始邮件 ------------------ 发件人: @.>; 发送时间: 2024年11月12日(星期二) 下午4:17 收件人: @.>; 抄送: @.***>; 主题: [fanyong920/jvppeteer] Cannot construct instance of com.ruiyun.jvppeteer.entities.Timestamp (Issue #159)

com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot construct instance of com.ruiyun.jvppeteer.entities.Timestamp (although at least one Creator exists): no double/Double-argument constructor/factory method to deserialize from Number value (1.731399248104643E12) at [Source: UNKNOWN; line: -1, column: -1] (through reference chain: com.ruiyun.jvppeteer.events.ExceptionThrownEvent["timestamp"]) at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:63) at com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1343) at com.fasterxml.jackson.databind.DeserializationContext.handleMissingInstantiator(DeserializationContext.java:1032) at com.fasterxml.jackson.databind.deser.ValueInstantiator.createFromDouble(ValueInstantiator.java:274) at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromDouble(StdValueInstantiator.java:378) at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromDouble(BeanDeserializerBase.java:1396) at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:175) at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:161) at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129) at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:288) at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:151) at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3984) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2276) at com.fasterxml.jackson.databind.ObjectMapper.treeToValue(ObjectMapper.java:2758) at com.ruiyun.jvppeteer.transport.CDPSession.lambda$onMessage$0(CDPSession.java:134) at com.ruiyun.jvppeteer.transport.Connection.lambda$startEmitEventThread$6(Connection.java:210) at java.lang.Thread.run(Thread.java:748) [WebSocketConnectReadThread-13] INFO com.ruiyun.jvppeteer.transport.WebSocketTransport - Connection closed by us Code: 1000 Reason: Normal closure

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: @.***>

coolguitar commented 1 day ago

我使用的版本是2.2.0，测试代码如下：

import java.util.ArrayList; import java.util.List; import java.util.function.Consumer;

import com.ruiyun.jvppeteer.common.Product; import com.ruiyun.jvppeteer.core.Browser; import com.ruiyun.jvppeteer.core.Page; import com.ruiyun.jvppeteer.core.Puppeteer; import com.ruiyun.jvppeteer.core.Request; import com.ruiyun.jvppeteer.core.Response; import com.ruiyun.jvppeteer.entities.GoToOptions; import com.ruiyun.jvppeteer.entities.LaunchOptions; import com.ruiyun.jvppeteer.entities.PuppeteerLifeCycle;

import dev.greenday.common.util.ConfigUtil;

public class Test {

public static void main(String[] args) throws Exception{
    LaunchOptions launchOptions =  LaunchOptions.builder().product(Product.CHROME).headless(false).devtools(false)
            .cacheDir("D:\\data\\chrome").args(getLaunchArguments()).build();

    Browser browser = Puppeteer.launch(launchOptions);
    Page page = browser.newPage();
    GoToOptions navigateOptions = new GoToOptions();
    List<PuppeteerLifeCycle> waitUntil = new ArrayList<>();
    waitUntil.add(PuppeteerLifeCycle.LOAD);
    navigateOptions.setWaitUntil(waitUntil);
    navigateOptions.setTimeout(1000*60*10);

    page.setRequestInterception(true);
    page.on(Page.PageEvent.Request, (Consumer<Request>) request -> {
        //System.out.println("response.url():" + request.url());
        if( request.url().contains("searchU")) {
            System.out.println("---------------------searchU.1");
            Response res = request.response();
            System.out.println("---------------------searchU.2");

            String data = new String(res.content());
            System.out.println(data);
        }
        request.continueRequest();
    });

// page.on(Page.PageEvent.Response, (Consumer) response -> { // System.out.println("response.url():" + response.url() +",header:" + String.valueOf(response.headers()));
// }); //login live page.goTo("https://teams.live.com/v2/",navigateOptions);

}

private static ArrayList<String> getLaunchArguments() {
    ArrayList<String> argList = new ArrayList<>();
    argList.add("--no-sandbox");
    argList.add("--disable-setuid-sandbox");
    //argList.add("--disable-blink-features=AutomationControlled");
    //argList.add("--auto-open-devtools-for-tabs");
    argList.add("--disable-infobars");
    //argList.add("--enable-webgl");
    //argList.add("--use-gl=swiftshader");
    argList.add("--disable-dev-shm-usage");
    argList.add("--lang=zh-CN");
    argList.add("--start-maximized");
    //设置插件
    argList.add("--disable-extensions-except=" + ConfigUtil.get("jvppeteer.chrome.plugin.path.nocookie"));
    argList.add("--load-extension=" + ConfigUtil.get("jvppeteer.chrome.plugin.path.nocookie"));
    return argList;
}

}

fanyong920 commented 1 day ago

ok,use 2.2.1  

樊勇 @.***

 

------------------ 原始邮件 ------------------ 发件人: @.>; 发送时间: 2024年11月12日(星期二) 下午4:39 收件人: @.>; 抄送: @.>; @.>; 主题: Re: [fanyong920/jvppeteer] Cannot construct instance of com.ruiyun.jvppeteer.entities.Timestamp (Issue #159)

我使用的版本是2.2.0，测试代码如下：

import java.util.ArrayList; import java.util.List; import java.util.function.Consumer;

import com.ruiyun.jvppeteer.common.Product; import com.ruiyun.jvppeteer.core.Browser; import com.ruiyun.jvppeteer.core.Page; import com.ruiyun.jvppeteer.core.Puppeteer; import com.ruiyun.jvppeteer.core.Request; import com.ruiyun.jvppeteer.core.Response; import com.ruiyun.jvppeteer.entities.GoToOptions; import com.ruiyun.jvppeteer.entities.LaunchOptions; import com.ruiyun.jvppeteer.entities.PuppeteerLifeCycle;

import dev.greenday.common.util.ConfigUtil;

public class Test { public static void main(String[] args) throws Exception{ LaunchOptions launchOptions = LaunchOptions.builder().product(Product.CHROME).headless(false).devtools(false) .cacheDir("D:\data\chrome").args(getLaunchArguments()).build(); Browser browser = Puppeteer.launch(launchOptions); Page page = browser.newPage(); GoToOptions navigateOptions = new GoToOptions(); List<PuppeteerLifeCycle> waitUntil = new ArrayList<>(); waitUntil.add(PuppeteerLifeCycle.LOAD); navigateOptions.setWaitUntil(waitUntil); navigateOptions.setTimeout(10006010); page.setRequestInterception(true); page.on(Page.PageEvent.Request, (Consumer<Request>) request -> { //System.out.println("response.url():" + request.url()); if( request.url().contains("searchU")) { System.out.println("---------------------searchU.1"); Response res = request.response(); System.out.println("---------------------searchU.2"); String data = new String(res.content()); System.out.println(data); } request.continueRequest(); });
// page.on(Page.PageEvent.Response, (Consumer) response -> { // System.out.println("response.url():" + response.url() +",header:" + String.valueOf(response.headers())); // }); //login live page.goTo("https://teams.live.com/v2/",navigateOptions); } private static ArrayList<String> getLaunchArguments() { ArrayList<String> argList = new ArrayList<>(); argList.add("--no-sandbox"); argList.add("--disable-setuid-sandbox"); //argList.add("--disable-blink-features=AutomationControlled"); //argList.add("--auto-open-devtools-for-tabs"); argList.add("--disable-infobars"); //argList.add("--enable-webgl"); //argList.add("--use-gl=swiftshader"); argList.add("--disable-dev-shm-usage"); argList.add("--lang=zh-CN"); argList.add("--start-maximized"); //设置插件 argList.add("--disable-extensions-except=" + ConfigUtil.get("jvppeteer.chrome.plugin.path.nocookie")); argList.add("--load-extension=" + ConfigUtil.get("jvppeteer.chrome.plugin.path.nocookie")); return argList; }
}

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

Knight-ZXW commented 1 day ago

一样的问题

coolguitar commented 1 day ago

使用2.2.1，有如下错误，2.2.0也同样存在，测试代码没有变化。

[JvEmitEventThread-2] ERROR com.ruiyun.jvppeteer.core.Request - request error: com.ruiyun.jvppeteer.exception.ProtocolException: Protocol error (method：Fetch.continueRequest): Invalid InterceptionId. at com.ruiyun.jvppeteer.transport.Callback.waitForResponse(Callback.java:56) at com.ruiyun.jvppeteer.transport.CallbackRegistry.create(CallbackRegistry.java:33) at com.ruiyun.jvppeteer.transport.Connection.rawSend(Connection.java:248) at com.ruiyun.jvppeteer.transport.CDPSession.send(CDPSession.java:81) at com.ruiyun.jvppeteer.transport.CDPSession.send(CDPSession.java:74) at com.ruiyun.jvppeteer.core.Request._continue(Request.java:339) at com.ruiyun.jvppeteer.core.Request.continueRequest(Request.java:309) at com.ruiyun.jvppeteer.core.Request.continueRequest(Request.java:467) at dev.greenday.contactus.ContactusPageRobot.lambda$0(ContactusPageRobot.java:801) at com.ruiyun.jvppeteer.core.Page.lambda$null$3(Page.java:211) at java.util.ArrayList.forEach(ArrayList.java:1257) at com.ruiyun.jvppeteer.core.Request.finalizeInterceptions(Request.java:514) at com.ruiyun.jvppeteer.core.NetworkManager.onRequest(NetworkManager.java:412) at com.ruiyun.jvppeteer.core.NetworkManager.onRequestPaused(NetworkManager.java:303) at com.ruiyun.jvppeteer.core.NetworkManager.lambda$addClient$0(NetworkManager.java:64) at com.ruiyun.jvppeteer.events.EventEmitter.emit(EventEmitter.java:80) at com.ruiyun.jvppeteer.transport.CDPSession.lambda$onMessage$0(CDPSession.java:134) at com.ruiyun.jvppeteer.transport.Connection.lambda$startEmitEventThread$6(Connection.java:210) at java.lang.Thread.run(Thread.java:748)

fanyong920 commented 1 day ago

我无法重现，你可以把出现错误的request的url和InterceptionId打印出来吗

coolguitar commented 1 day ago

错误比较多，截取几个如下：

[JvEmitEventThread-1] ERROR com.ruiyun.jvppeteer.core.Request - request error: com.ruiyun.jvppeteer.exception.ProtocolException: Protocol error (method：Fetch.continueRequest): Invalid InterceptionId. at com.ruiyun.jvppeteer.transport.Callback.waitForResponse(Callback.java:56) at com.ruiyun.jvppeteer.transport.CallbackRegistry.create(CallbackRegistry.java:33) at com.ruiyun.jvppeteer.transport.Connection.rawSend(Connection.java:248) at com.ruiyun.jvppeteer.transport.CDPSession.send(CDPSession.java:81) at com.ruiyun.jvppeteer.transport.CDPSession.send(CDPSession.java:74) at com.ruiyun.jvppeteer.core.Request._continue(Request.java:339) at com.ruiyun.jvppeteer.core.Request.continueRequest(Request.java:309) at com.ruiyun.jvppeteer.core.Request.continueRequest(Request.java:467) at dev.greenday.contactus.Test.lambda$0(Test.java:50) at com.ruiyun.jvppeteer.core.Page.lambda$null$3(Page.java:211) at java.util.ArrayList.forEach(ArrayList.java:1257) at com.ruiyun.jvppeteer.core.Request.finalizeInterceptions(Request.java:514) at com.ruiyun.jvppeteer.core.NetworkManager.onRequest(NetworkManager.java:412) at com.ruiyun.jvppeteer.core.NetworkManager.onRequestWillBeSent(NetworkManager.java:248) at com.ruiyun.jvppeteer.core.NetworkManager.lambda$addClient$2(NetworkManager.java:72) at com.ruiyun.jvppeteer.events.EventEmitter.emit(EventEmitter.java:80) at com.ruiyun.jvppeteer.transport.CDPSession.lambda$onMessage$0(CDPSession.java:134) at com.ruiyun.jvppeteer.transport.Connection.lambda$startEmitEventThread$6(Connection.java:210) at java.lang.Thread.run(Thread.java:748) request.url():https://statics.teams.cdn.live.net/teams-modular-packages/hashed-assets/70126-39880cf8e0e778bd.js,request.interceptionId():interception-job-232.0 [JvEmitEventThread-1] ERROR com.ruiyun.jvppeteer.core.Request - request error: com.ruiyun.jvppeteer.exception.ProtocolException: Protocol error (method：Fetch.continueRequest): Invalid InterceptionId. at com.ruiyun.jvppeteer.transport.Callback.waitForResponse(Callback.java:56) at com.ruiyun.jvppeteer.transport.CallbackRegistry.create(CallbackRegistry.java:33) at com.ruiyun.jvppeteer.transport.Connection.rawSend(Connection.java:248) at com.ruiyun.jvppeteer.transport.CDPSession.send(CDPSession.java:81) at com.ruiyun.jvppeteer.transport.CDPSession.send(CDPSession.java:74) at com.ruiyun.jvppeteer.core.Request._continue(Request.java:339) at com.ruiyun.jvppeteer.core.Request.continueRequest(Request.java:309) at com.ruiyun.jvppeteer.core.Request.continueRequest(Request.java:467) at dev.greenday.contactus.Test.lambda$0(Test.java:50) at com.ruiyun.jvppeteer.core.Page.lambda$null$3(Page.java:211) at java.util.ArrayList.forEach(ArrayList.java:1257) at com.ruiyun.jvppeteer.core.Request.finalizeInterceptions(Request.java:514) at com.ruiyun.jvppeteer.core.NetworkManager.onRequest(NetworkManager.java:412) at com.ruiyun.jvppeteer.core.NetworkManager.onRequestWillBeSent(NetworkManager.java:248) at com.ruiyun.jvppeteer.core.NetworkManager.lambda$addClient$2(NetworkManager.java:72) at com.ruiyun.jvppeteer.events.EventEmitter.emit(EventEmitter.java:80) at com.ruiyun.jvppeteer.transport.CDPSession.lambda$onMessage$0(CDPSession.java:134) at com.ruiyun.jvppeteer.transport.Connection.lambda$startEmitEventThread$6(Connection.java:210) at java.lang.Thread.run(Thread.java:748) request.url():https://statics.teams.cdn.live.net/teams-modular-packages/hashed-assets/602438-e94806bc95e51c6b.js,request.interceptionId():interception-job-233.0 [JvEmitEventThread-1] ERROR com.ruiyun.jvppeteer.core.Request - request error: com.ruiyun.jvppeteer.exception.ProtocolException: Protocol error (method：Fetch.continueRequest): Invalid InterceptionId. at com.ruiyun.jvppeteer.transport.Callback.waitForResponse(Callback.java:56) at com.ruiyun.jvppeteer.transport.CallbackRegistry.create(CallbackRegistry.java:33) at com.ruiyun.jvppeteer.transport.Connection.rawSend(Connection.java:248) at com.ruiyun.jvppeteer.transport.CDPSession.send(CDPSession.java:81) at com.ruiyun.jvppeteer.transport.CDPSession.send(CDPSession.java:74) at com.ruiyun.jvppeteer.core.Request._continue(Request.java:339) at com.ruiyun.jvppeteer.core.Request.continueRequest(Request.java:309) at com.ruiyun.jvppeteer.core.Request.continueRequest(Request.java:467) at dev.greenday.contactus.Test.lambda$0(Test.java:50) at com.ruiyun.jvppeteer.core.Page.lambda$null$3(Page.java:211) at java.util.ArrayList.forEach(ArrayList.java:1257) at com.ruiyun.jvppeteer.core.Request.finalizeInterceptions(Request.java:514) at com.ruiyun.jvppeteer.core.NetworkManager.onRequest(NetworkManager.java:412) at com.ruiyun.jvppeteer.core.NetworkManager.onRequestWillBeSent(NetworkManager.java:248) at com.ruiyun.jvppeteer.core.NetworkManager.lambda$addClient$2(NetworkManager.java:72) at com.ruiyun.jvppeteer.events.EventEmitter.emit(EventEmitter.java:80) at com.ruiyun.jvppeteer.transport.CDPSession.lambda$onMessage$0(CDPSession.java:134) at com.ruiyun.jvppeteer.transport.Connection.lambda$startEmitEventThread$6(Connection.java:210) at java.lang.Thread.run(Thread.java:748) request.url():https://statics.teams.cdn.live.net/teams-modular-packages/hashed-assets/602545-bdefe7113e2ef1aa.js,request.interceptionId():interception-job-234.0

fanyong920 commented 20 hours ago

我似乎观察到了问题所在。所有这些报错的Request，在页面已经取消了这些Request，所以显示Invalid InterceptionId. 错误，这些错误在 Puppeteer中是容忍的，但是会打印日志，你不必理会。

至于为什么会取消Request,可能是由于重定向的原因，这是我猜测的。取消Requrst是页面的行为。

coolguitar commented 18 hours ago

好的，谢谢~~