Ahacad / FDUCourseData

复旦大学本科开课数据爬虫及历年开课数据
8 stars 3 forks source link

Failed to make http request #3

Closed Ahacad closed 3 years ago

Ahacad commented 3 years ago

When doing yarn jwfw, I have set the SEMESTER_ID to 364 and the whole JWFW_COOKIES with cookie, JSESSIONID, HM_lvt_2358..., etc.

(node:153977) UnhandledPromiseRejectionWarning: Error: Client network socket disconnected before secure TLS connection was established
    at connResetException (internal/errors.js:607:14)
    at TLSSocket.onConnectEnd (_tls_wrap.js:1554:19)
    at TLSSocket.emit (events.js:388:22)
    at TLSSocket.emit (domain.js:470:12)
    at endReadableNT (internal/streams/readable.js:1336:12)
    at processTicksAndRejections (internal/process/task_queues.js:82:21)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:153977) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)
(node:153977) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
Done in 1.47s.
Ahacad commented 3 years ago

By the way, it seems a good idea to add .catch after making axios requests.

CLDXiang commented 3 years ago

By the way, it seems a good idea to add .catch after making axios requests.

Maybe use an axois interceptor

CLDXiang commented 3 years ago

请求失败是因为选课没开吧?

Ahacad commented 3 years ago

不是把,jwfw 开了的呀

CLDXiang commented 3 years ago

看起来是 jwfw API 改了,从 POST 改成 GET 了,现在用 URL Query 而非 Form Data 传参数, @Ahacad 你打算试试修一下吗?

Ahacad commented 3 years ago

emmmm,等我考完把😅,本来复习累了想试试一个小时里面能不能搞定,现在看来还是等 7.1 后 :sweat_smile:

CLDXiang commented 3 years ago

ok

Ahacad commented 3 years ago

我看了一下貌似没有变 GET? 20210622-15:25:43

Ahacad commented 3 years ago

要不 @CLDXiang 试试能不能跑起来,我怀疑是这里代理的问题

Ahacad commented 3 years ago

好的应该就是我代理问题,axios 里面加一个选项 proxy: false 取消自动检测当前环境的代理

之后就爬取成功

CLDXiang commented 3 years ago

image

ok,目前看起来是 POST 和 GET API 同时存在

Ahacad commented 3 years ago

诶我搜索的时候没有这个 GET 的请求。。