Part files are filling up the Hard disk

[Alexxino]

Hello,

When the clients are sending the parts, the server is merging them into a savework file, however all the temp parts are there forever filling up the HDD, is there any option to autodelete the parts when they are merged ?

Can we trigger the merger more often?

[Etayson]

All merged files with .part extention should be deleted after merging (source line 2072). Even file with crc error should be deleted (source line 1676). Show me your parametrs for server and client. Also tell me you are used released version or compiled by yourself ?

[Alexxino]

This is my configuration: Server: EtarkangarooServer.exe -checkdp -dp 16 -wi 100 -beginrange 200000000000000000000000000000000 -endrange 3ffffffffffffffffffffffffffffffff -pub 03633cbe3ec02b9401c5effa144c5b4d22f87940259634858fc7e59b1c09937852

I have 2 Clients: Client 1: EtarkangarooClient.exe -pool 127.0.0.1:8000 -d 0,1 -grid 176,256,176,256 -wi 200 -wmerge -min 10

Client 2: EtarkangarooClient.exe -pool 192.168.1.12:8000 -d 0,1 -grid 224,256,160,256 -wi 350 -wmerge -min 10

I'm not sure whether the problem is the Harddisk that cannot handle too much I/O at the same time or what but I have more than 20 part files of 200-300 MB each

[Etayson]

First of all remove -wmerge flag. In that case every work file on client side will have different names while saving. Second for 130 bit range -dp 16 is toooooo low, you will need keep huge amount of TB on HDD. Try at least 27 or so.

Upd. Seems like you have weak CPU on server side, so remove flag -checkdp. In that case working files wil be merged to savework file without checking.

[Alexxino]

Thank you for your response. Do I need to manually merge all the files of the clients by removing the -wgmerge flag?? If the work of the clients is not merged then we dont know if we have collision, right ?

[Etayson]

All .part files on server side these are files that have not yet been merged. Stop the server, remove flag -checkdp from server parameters. Start server. After that, all .part files will be sequentially merged automatically.

Of course, until the files are merged, you will not know if there was a collision. There is a case th at the collision occurs on the client side, this is also possible. Because the client continues the path and has a part of the hashtable before saving. But this is rare. So you need to remove the flag -checkdp on server side to speed up the processing of files from clients. Also remove -wmerge flag on client side to save files with different names if you have problems with CRC.

And i recomended to you change -dp (16 is too small for 130 bits, you would need thousands of terabytes of disk space to hold the entire table.) but in this case your curent working file will be useless.

Upd. If you decide to change -dp, then you should also change the -wi parameter on clients. It makes no sense to save files every 200-300 seconds. Set at least 3-6 hours (10800-21600).

[Alexxino]

Yeah that is what I have done thank you.

I have this new scenario:

• Server without checkdp
• Clients without -wmerge

My question is the following: how the server knows if there is a collision? All the client work files are not together in the server (they are not sent to the server)

[Etayson]

What do you mean by "Not sent to the server"? Each time an interval equal to the -wi parameter passes. The Kangaroo program saves the hashtable to a file. And the client part sends this file to the server. And on the server side, all client files are combined into one. With checking for collisions when merging.

Before using programs on such a huge range as 130 bits, try using programs on a small range to understand how it works. For example, 75 bits.

[Alexxino]

Here you can see that the -wi parameter is passed and also the kangaroo file is saved but not sent to the server I run the client with: EtarkangarooClient.exe -pool 127.0.0.1:8000 -d 0,1 -grid 176,256,176,256 -wi 100 -min 10 -wmerge And the server with: EtarkangarooServer.exe -dp 26 -wi 280 -beginrange 200000000000000000 -endrange 3fffffffffffffffff -pub 0290e6900a58d33393bc1097b5aed31f2e4e7cbd3e5466af958665bc0121248483

I'm trying now with puzzle 70

[Etayson]

You didn't understand the parameters well enough. Usage:

-wi       [optional] timer interval for autosaving ht/kangaroos, without setting, the value from the server is used
-min      [optional] the minimum size (in MB) of a working file to send to the server
-name     [optional] name of client
-pool     [required] IP adress of server:port, default 127.0.0.1:8000
-grid     [Etarkangaroo settings]
-d        [Etarkangaroo settings]
-wmerge   [Etarkangaroo settings]

For puzzle 70 you not need such big -dp size. For 70bit set -dp 16 You also set -wi 100 which means saving every 100 seconds, but at the same time you set the -min 10 flag which allows sending files only larger than 10MB. Look to your screenshot.. Your file is 0.10MB becouse you set big -dp size. And so the client program will merge the file 100 more times on the client side before the file will larger than 10MB and it is sent to the server. If you set dp to 16 bits, then the file would weigh enough to send it to the server. Set -dp 16 on server side. Remove -min flag on client side. An retry.

Upd. here is logic how programms work with your current configration. Etarkangaroo save hashtable every 100s (-wi 100) Client app look that you set -min 10 wait while file grow to 10MB size. Due to using -wmerge flag Etarkangaroo will merge every hashtable to single file on client side. Once file grow to 10MB client app send it to server.

Even if you remove flag -min the minimum size for sending file will be 1MB So you need decrease -dp to 16

[Alexxino]

Alright, I understand it now!

Thank you so much !

PD: let me know if you can successfully compile a client for linux, that would be awesome in order to use it with renting Graphic cards which are always on linux systems.

[Alexxino]

Hello @Etayson I have been trying the puzzle 130 with 26 DP size for some hours but the clients are not sending the files even when they are more than 10MB.

Server :

The client doesnt have -wmerge flag. Is anything I can do to force the merge?

[Etayson]

Look to the screenshot of ther server app. You freeze cmd. Select server app window and press esc.

[Alexxino]

No, it is not frozen. I have closed it and run it again and its the same. I think problem is with clients.

[Etayson]

At first glance, the problem is that you did not delete the old files that are less than 10MB after you change -dp Stop client. Delete files ...._test1 that less than 10MB. Restart client. Let me know result. Client app look to the folder and search any files where _test1 present. So first file that client app took is 08_11_2023_16_01_04_test1 but it turned out to be less than the minimum size. Therefore, the client app simply waits for the file to grow and does not move on to other files.

[Alexxino]

That is not correct. Before running the program (client and server) there were no files. I will do what you said but do I lose the progress ?

[Etayson]

If you always run server with the same -dp size and client with the same -wi (dependency on how many GPUs) will produce file almost the same size. in your screenshot there files that more than 10MB and files that have size 96-103KB. That mean that you run different configuration (change -dp size or -wi). Delete small files _test1 files (this file have minimal progress) If you don't delete the old small files, the client will get stuck on this and wait until the first file grows to the minimum size.

[Alexxino]

You were totally right, after deleting the small test1 files, the client has started to send all the files. If I find the solution I will reward you for your program. @Etayson Congrats for the software.

PD: When I find enough time I will try to compile the software for Linux x64 and I will let you know. Do you have any other way of communication? telegram maybe?

[Etayson]

Did you made calculation how many disk space you need for -dp 26 and how many years you need to solve puzzle #130 with your current hash power? For -dp 26 Expected operations 2^65.55 Expected HT size: 29.37 TB Your current hashpower 1348 Mk/s =2^30.3281 Expected time 2^65.55 / 2^30.3281 = 2^35.2219s ~463800 days ~ 1270 years

[Alexxino]

True, but there is also the lucky factor ...

I have tried to compile your program to linux but the free version is limited to compile 800 lines... so it is not possible, any chance that you try to compile it ? the installation in linux is quite easy.

[Etayson]

For kangaroo lucky factor is minimal. Look https://github.com/JeanLucPons/Kangaroo/blob/master/DOC/successprob.jpg Even if you very-very lucky and you solve puzzle at 5% you need do 0.5*sqrt(N) operations = 2^63.5 it is 111000 days.

I do not dissuade you, but I want to warn you that solving puzzle 130 is not an easy task. This is not the same as solving puzzle 70. Run, wait, done. This is a huge job both to keep hundreds and hundreds of GPUs in operation. You also need a very fast disk (nvme) to hold the hashtable. Otherwise, over time, merging files will take a huge amount of time. You also need to back up this table. Have a plan for how to work with the table when it becomes huge and each file merging with the table will cost you a disk resource. And in the end, accept the possibility that someone will find you faster and all your efforts and spent resources will be in vain. And of course, it's up to you to decide.

[Alexxino]

Thank you so much for showing this information, honestly I didn’t know it was such a big number. I think I have more real chances trying the puzzle 66 using your rotor cuda