Closed ISJ-439 closed 2 years ago
@ISJ-439 Thank you for your bug report. Unfortunately I don't understand it.
Please answer to the following questions:
Hello,
Sorry it's not clear. I also wanted to say, the only reason why I dont use the built in search is due tot he sheer volume of the library, its just too large at 990GB and 16 separate ZIM files. Yacy is also a lot better at fuzzy searching.
What is the version of kiwix-serve you use? Which system? 3.1.2
What is the detailed step-by-step procedure to reproduce the bug?
If you we're to do this by yourself:
sudo apt install kiwix kiwix-tools docker.io
sudo mkdir /opt/yacy_data sudo chmod 777 /opt/yacy_data
Select: Access only with qualified account Peer User: adminuser Set the passwords
Select: Robinson Mode Select: Private Peer
The maximum size of the cache: 50MB Compression level: 0
Max searches in 3s: 3 Max searches in 1mn: 30 Max searches in 10mn: 300
Greeting Line: Search the archived copies of Wikipedia for removed or changed articles. URL of Home Page: http://example.com:8090/ Index remote results: uncheck (this system is for searching the crawled pages only)
sudo docker run -d --name yacy -p 8090:8090 -p 8443:8443 -v /opt/yacy_data:/opt/yacy_search_server/DATA --log-opt max-size=200m --log-opt max-file=2 yacy/yacy_search_server:latest
sudo adduser kiwixuser
su - kiwixuser -c 'kiwix-serve --library /home/kiwixuser/example.zim --port 5555 --nosearchbar --daemon'
>What do you get?
An error 503 on a crawl but it works fine from
>What do you expect?
It to crawl the zim file and index the text within for later searches.
To me its sounds this should be a bug report for Yacy, which I have no clue about. If everything works fine without Yacy, then it should be investigated in that direction. The best I can imagine is that the Yacy is flooding the server which at some point is not able to answer properly anymore because of lack of resources. The fulltext search can be pretty resource intensive in particular. To start investigation I should better have a reproduction case without Yacy.
Thanks, the fact it's returning a 503 is why I posted it here, as that's typically a server related error.
@ISJ-439 Secure that your crawler does not make more than one request per second. If you still have errors, share the corresponding log please.
@kelson42 That may be it, any other anti-abuse settings to avoid?
No, you can use the "--threads" option to increase the througput.
For future googles, this was an issue with other software parsing the pages without adjusting the custom ports while doing so.
@ISJ-439 Without putting in question your choice of Yaci to make fulltext searches, feel free to open a ticket or two if the current fulltext search engine (Xapian engine) misses a specific feature. We are not over with improving Kiwix search capabilities and user articulated feedbacks are therefore very valuable.
@ISJ-439 Without putting in question your choice of Yaci to make fulltext searches, feel free to open a ticket or two if the current fulltext search engine (Xapian engine) misses a specific feature. We are not over with improving Kiwix search capabilities and user articulated feedbacks are therefore very valuable.
Done sir/miss: https://github.com/kiwix/kiwix-tools/issues/528
Hello,
When loading a new page to crawl, which is a locally hosted page on the same server, I'm getting a error 503. However using Lynx, a cli web browser for linux I can load it fine.
How would i troubleshoot this further?
Yacy Console: https://i.imgur.com/IwpjkuS.png
Kiwix Server Logs: