platonai / exotic-amazon

A complete solution to crawl amazon at scale completely and accurately.
146 stars 47 forks source link

修改mongoDB配置不成功 #40

Open xiaoer666 opened 11 months ago

xiaoer666 commented 11 months ago

我再application文件种修改mongodb配置项,但是运行后日志中仍有尝试连接默认mongoDB

配置为

gora.mongodb.override_hadoop_configuration=true
gora.mongodb.servers=127.0.0.1:17017
gora.mongodb.login=root
gora.mongodb.secret=xxxxxxxx

spring.data.mongodb.uri=mongodb://root:xxxxxxxx@127.0.0.1:17017/scent

日志中关键信息:

2023-11-11 20:40:45.774  INFO [main] o.m.d.cluster - Cluster created with settings {hosts=[127.0.0.1:17017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms'}
2023-11-11 20:40:45.940  INFO [.0.1:17017] o.m.d.connection - Opened connection [connectionId{localValue:2, serverValue:10}] to 127.0.0.1:17017
2023-11-11 20:40:45.938  INFO [.0.1:17017] o.m.d.connection - Opened connection [connectionId{localValue:1, serverValue:9}] to 127.0.0.1:17017
2023-11-11 20:40:45.941  INFO [.0.1:17017] o.m.d.cluster - Monitor thread successfully connected to server with description ServerDescription{address=127.0.0.1:17017, type=STANDALONE, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=21, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=56057300}

2023-11-11 20:40:53.328  INFO [r-worker-1] a.p.s.o.m.d.cluster - Cluster created with settings {hosts=[127.0.0.1:27017], mode=MULTIPLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=500}
2023-11-11 20:40:53.328  INFO [r-worker-1] a.p.s.o.m.d.cluster - Adding discovered server 127.0.0.1:27017 to client view of cluster
2023-11-11 20:40:53.382  INFO [.0.1:27017] a.p.s.o.m.d.cluster - Exception in monitor thread while connecting to server 127.0.0.1:27017
2023-11-11 20:41:24.655  WARN [r-worker-6] a.p.p.p.WebDb - Data storage failure | [get] - Timed out after 30000 ms while waiting to connect. Client view of cluster state is {type=UNKNOWN, servers=[{address=127.0.0.1:27017, type=UNKNOWN, state=CONNECTING, exception={ai.platon.shaded.com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused: connect}}]
2023-11-11 20:41:24.655  WARN [r-worker-5] a.p.p.p.WebDb - Data storage failure | [get] - Timed out after 30000 ms while waiting to connect. Client view of cluster state is {type=UNKNOWN, servers=[{address=127.0.0.1:27017, type=UNKNOWN, state=CONNECTING, exception={ai.platon.shaded.com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused: connect}}]
2023-11-11 20:41:24.655  WARN [-worker-11] a.p.p.p.WebDb - Data storage failure | [get] - ai.platon.shaded.com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting to connect. Client view of cluster state is {type=UNKNOWN, servers=[{address=127.0.0.1:27017, type=UNKNOWN, state=CONNECTING, exception={ai.platon.shaded.com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused: connect}}]
xiaoer666 commented 11 months ago

似乎找到了一点线索,在package ai.platon.exotic.amazon.starter下的CrawlStart中找到以下代码

        if (alwaysFalse() && NetUtil.testNetwork("127.0.0.1", 28018)) {
            // TODO: find out why the settings in application.properties do not work
            conf.unbox().set("gora.mongodb.override_hadoop_configuration", "false")
            conf.unbox().set("gora.mongodb.servers", "127.0.0.1:28018")
        }
xiaoer666 commented 11 months ago

但似乎并没有关系,我在里面手动把我的mongoDB配置上还是出现一样的问题

        if (alwaysFalse() && NetUtil.testNetwork("127.0.0.1", 17017)) {
            // TODO: find out why the settings in application.properties do not work
            conf.unbox().set("gora.mongodb.override_hadoop_configuration", "false")
            conf.unbox().set("gora.mongodb.servers", "127.0.0.1:17017")
            conf.unbox().set("gora.mongodb.root", "root")
            conf.unbox().set("gora.mongodb.secret", "xxxxxxx")
        }
galaxyeye commented 9 months ago

remove alwaysFalse()

ZhujingJava commented 7 months ago

1.我在调试Mongo的时候同样遇到这个问题,去掉alwaysFalse()也没有用,还是会提示 2024-03-07 15:46:47.044 WARN [r-worker-7] a.p.p.p.WebDb - Data storage failure | [get] - Timed out after 30000 ms while waiting to connect. Client view of cluster state is {type=UNKNOWN, servers=[{address=127.0.0.1:27017, type=UNKNOWN, state=CONNECTING, exception={ai.platon.shaded.com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused: connect}}] 我的yaml文件配置如下: gora.mongodb.override_hadoop_configuration=false gora.mongodb.servers=192.168.1.252:27017 spring.data.mongodb.uri=mongodb://192.168.1.252:27017/scent 2.第一次有提示连接 2024-03-07 15:52:26.990 INFO [tartedMain] o.m.d.cluster - Cluster created with settings {hosts=[192.168.1.252:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms'}

3.随后提示 2024-03-07 15:52:35.672 INFO [r-worker-1] a.p.s.c.AbstractRunnableCrawler - Injecting seeds ... 2024-03-07 15:52:35.672 INFO [r-worker-1] a.p.s.c.AbstractRunnableCrawler - Generating fetch urls ... 2024-03-07 15:52:35.840 INFO [r-worker-1] a.p.s.o.m.d.cluster - Cluster created with settings {hosts=[127.0.0.1:27017], mode=MULTIPLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=500} 2024-03-07 15:52:35.840 INFO [r-worker-1] a.p.s.o.m.d.cluster - Adding discovered server 127.0.0.1:27017 to client view of cluster 2024-03-07 15:52:35.864 INFO [.0.1:27017] a.p.s.o.m.d.cluster - Exception in monitor thread while connecting to server 127.0.0.1:27017 4.还是没找到这个 127.0.0.1:27017 出自何处,我发现第一次连接成功走的是o.m.d.cluster 路径,这个应该是官方的以来,第二次连接失败走的是a.p.s.o.m.d.cluster这个应该是作者的依赖包。 大佬帮忙解答一下