yacy / yacy_search_server

Distributed Peer-to-Peer Web Search Engine and Intranet Search Appliance
http://yacy.net
Other
3.39k stars 427 forks source link

Crawling 18 sites to depth of 5 stops yacy and logging info. Minor work in progress to remove hazelcast lib #504

Closed smokingwheels closed 2 years ago

smokingwheels commented 2 years ago

`I 2022/09/28 19:07:34 com.hazelcast.internal.diagnostics.HealthMonitor [192.168.1.20]:5701 [YaCyP2P] [4.2] processors=8, physical.memory.total=16.0G, physical.memory.free=8.1G, swap.space.total=0, swap.space.free=0, heap.memory.used=518.6M, heap.memory.free=81.4M, heap.memory.total=600.0M, heap.memory.max=600.0M, heap.memory.used/total=86.44%, heap.memory.used/max=86.44%, minor.gc.count=8289, minor.gc.time=32418ms, major.gc.count=1321, major.gc.time=126135ms, load.process=0.00%, load.system=9.51%, load.systemAverage=n/a thread.count=339, thread.peakCount=656, cluster.timeDiff=1989, event.q.size=0, executor.q.async.size=0, executor.q.client.size=0, executor.q.client.query.size=0, executor.q.client.blocking.size=0, executor.q.query.size=0, executor.q.scheduled.size=0, executor.q.io.size=0, executor.q.system.size=0, executor.q.operations.size=0, executor.q.priorityOperation.size=0, operations.completed.count=30679, executor.q.mapLoad.size=0, executor.q.mapLoadAllKeys.size=0, executor.q.cluster.size=0, executor.q.response.size=0, operations.running.count=0, operations.pending.invocations.percentage=0.00%, operations.pending.invocations.count=0, proxy.count=1, clientEndpoint.count=0, connection.active.count=0, client.connection.count=0, connection.count=0

I 2022/09/28 19:07:33 YACY * publish: handshaked senior peer 'agent-gafumaf-ufe-45' at [217.83.192.73]`

Was crawling 18 sites to depth of 5 with 600 mb of memory for a little while as a test.

Windows 11 old I7 Java 19 HDD When crawling my wifi was pulling approx 16 mega bits a second of data.

smokingwheels commented 2 years ago

Crawler is paused with green button on the pause. Click it and yacy starts to crawl again. Yes it ran out of memory again. Got to 92% used.

Orbiter commented 2 years ago

hazelcast is removed