search

yacy / yacy_search_server

Distributed Peer-to-Peer Web Search Engine and Intranet Search Appliance
http://yacy.net
Other
3.38k stars 427 forks source link

Please review my changes. #525

Open smokingwheels opened 1 year ago

smokingwheels commented 1 year ago

Please review my changes. Possible startYACY.bat back to javaw.

https://github.com/yacy/yacy_search_server/compare/master...smokingwheels:yacy_search_server:master

minecraftchest1 commented 1 year ago

The proper thing to do is to open a pull request.

smokingwheels commented 1 year ago

I don't want to mess it up...There are bigger files. to include java in the git. Not sure how much room there is. Able to merge. These branches can be automatically merged.

Orbiter commented 1 year ago

Well a lot of changes but anything that helps to get the windows version back on the road helps. So what is achieved here?

smokingwheels commented 1 year ago

I want to add Java 11 and Java 19 binary source files with the Windows install tools which I have written in QB64 some 5 months ago. It will make it easier to publish changes in the Yacy development from a separate precompiled repository for windows and linux. See this for testing. https://github.com/smokingwheels/YaCy.

The main thing I am concerned about is I need to add approx 350 MB's of files in the main yacy git repository and if this will cause any issues in the size requirements for the current repository? The current 350 MB's can be cut down to just one version of Java if needed.

Orbiter commented 1 year ago

where are the files from the RELEASEJAVA path come from? we need a license statement there. I wonder if it would be possible to load those files during build phase from somewhere else. It is indeed not so good to put it into the git repo. With a proper licensing we could also host them ourselves.

smokingwheels commented 1 year ago

need a license statement there. Ok

RELEASEJAVA path come from? Maybe possible to use from RedHat java-11-openjdk-11.0.16.1.1-1.windows.redhat.x86_64.msi. But its 255 Mb's, will try soon.

The others are from Oracle America, Inc. and Microsoft. Hosting ok I think I have an EXE file.

Orbiter commented 1 year ago

ok so here we have a problem, if those files are from Oracle and/or Microsoft, they are not free software! The only way to use those file is to pull them during installation process, not even during packaging!

Is the another option to use temurin from adoptium?

smokingwheels commented 1 year ago

temurin from adoptium

I Think I tried it and found that on batch file install the Java Home path is not set.

pull them during installation process

There is Url2File I have used many years ago. http://www.chami.com/free/url2file_wincon.html Will look into that.

Orbiter commented 1 year ago

I'm sorry to say that Url2File has also no license information and no source code anywhere, we also cannot use that :(

smokingwheels commented 1 year ago

ok find another method.

I will look into writing something in QB64 see https://github.com/smokingwheels/Yacy_front_end I am learning to code the web server in it. From The current QB64 source https://github.com/DualBrain/QB64 I only use it to generate my code not to embed QB64 as a whole Program.

Its MIT License. Copyright (c) 2022 Cory Smith

Are the *.exe files and source code I create abled to be used to pull java source from?

frankenstein91 commented 1 year ago

@smokingwheels why do you choose to deny files like zip, pdf and sound files from the search? https://github.com/yacy/yacy_search_server/compare/master...smokingwheels:yacy_search_server:master#diff-7aa9b44bec22b8d6558031520ac664ae0ff4b8b3c96de740599842e287c412a8L331

smokingwheels commented 1 year ago

@frankenstein91 I only have a slow NBN internet connection, because my line is over 1 km long to the NBN fiber box.

deny files like zip, pdf and sound

Reference: Internet Usage: I have done ~1.8 TB of traffic a month with current configurations crawling 24 hours a day before the changes.

The reasons. The Crawler Que Loader gets stuck holding on and downloading all the files by the looks. Yacy starts out crawling at approx 1000 ppm on my devices, then slows down when there are to many in the list.
If you wait they will drop off the list. Its can be as high as 80-120 URLs.

My collection of URLs from IndexCreateLoaderQueue_p.html. http://gts.undo.it/index.php/s/6w5RfgjSMfRiWiG

I only have at least 12 year old recycled devices to use.

I have a video of after the mods to deny files ~2000 PPM peak. https://www.youtube.com/watch?v=iZKbjlmygQ0

Hope that helps.

ArneBab commented 1 year ago

Maybe you can pull the Java releases from Adoptium. The Freenet / Hyphanet installer bundles Java 11: https://github.com/hyphanet/wininstaller-innosetup/tree/master/install_bundle

smokingwheels commented 12 months ago

I still have a lot to sort out... I have tried windows advanced installer and there's an error but it generates a Mac os app.

https://community.searchlab.eu/t/yacy-v1-930-windows-and-linux-builds