search

VI4IO / io-500-dev

IO-500
37 stars 22 forks source link

find phase: bash version doesn't look at named parameters in .ini file #70

Closed carlilek closed 4 years ago

carlilek commented 4 years ago

The bash version of the benchmark appears to ignore the parameters under the [find] section of the config.ini file, namely pfind-steal-next pfind-queue-length pfind-parallelize-single-dir-access-using-hashing nproc

The last of them would probably require a fair amount of extra code to reinterpret the nproc requested to new mpirun_args.

Evidence: `[find]

Pfind Steal from next

pfind-steal-next = TRUE

Pfind queue length

pfind-queue-length = 20000

Pfind with hashing

pfind-parallelize-single-dir-access-using-hashing = TRUE

Set the number of processes for pfind

nproc = 900

`

from bash script output: [Starting] find [Exec] mpiexec -np 300 -ppn 30 -f /io500/io500-isc20/io500-app/hostfile-10-h09 --bootstrap ssh /io500/io500-isc20/io500-app/bin/pfind ./datadir/2020.06.01-13.50.21-scr -newer ./datadir/2020.06.01-13.50.21-scr/timestampfile -size 3901c -name "*01*"

from C script: [find] t_start = 2020-06-01 14:11:03 exe = ./pfind ./datadir/2020.06.01-13.50.21-app -newer ./datadir/2020.06.01-13.50.21-app/timestampfile -size 3901c -name *01* -C -N -H 1 -q 20000

JulianKunkel commented 4 years ago

Could you fill them into the "external-extra-args" section? This relates to #66, an unofficial bugfix is available in the app master.

carlilek commented 4 years ago

Yes, I suspect that would be a functional temporary workaround. I'm not seeing exactly where the parameters would need to be translated for the bash version (I think the io500.sh calls io500-fixed.sh directly, so I guess it would need to be there?)

On Jun 1, 2020, at 2:35 PM, Julian Kunkel notifications@github.com<mailto:notifications@github.com> wrote:

Could you fill them into the "external-extra-args" section? This relates to #66https://urldefense.com/v3/__https://github.com/VI4IO/io-500-dev/issues/66__;!!Eh6p8Q!UasW6FyNSVH0TIb2AvqewXBMTqXLvMYyL6UyIEiU6PYboe-6KX8T9MShm8VTa-Xt9LzM$, an unofficial bugfix is available in the app master.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/VI4IO/io-500-dev/issues/70*issuecomment-637033112__;Iw!!Eh6p8Q!UasW6FyNSVH0TIb2AvqewXBMTqXLvMYyL6UyIEiU6PYboe-6KX8T9MShm8VTa7HVlsn7$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AC3AFHIVZWO56NTHI2YNONLRUPYFLANCNFSM4NQBJDLA__;!!Eh6p8Q!UasW6FyNSVH0TIb2AvqewXBMTqXLvMYyL6UyIEiU6PYboe-6KX8T9MShm8VTa-aMwZJb$.

JulianKunkel commented 4 years ago

Basically, the external-extra-args would then support: pfind-steal-next pfind-queue-length pfind-parallelize-single-dir-access-using-hashing

However, nproc won't be supported at all, in #66, I wrote that spawning a bash from an MPI process that executes MPI doesn't necessarily work. With current OpenMPI it doesn't anymore. Therefore, needs a changed/new pfind using the same API.

johnbent commented 4 years ago

Can't you write a wrapper to pfind that is called by just rank 0 and it itself spawns a new MPI job?

From: Julian Kunkel notifications@github.com Sent: Monday, June 1, 2020 12:48 PM To: VI4IO/io-500-dev io-500-dev@noreply.github.com Cc: Subscribed subscribed@noreply.github.com Subject: Re: [VI4IO/io-500-dev] find phase: bash version doesn't look at named parameters in .ini file (#70)

Basically, the external-extra-args would then support: pfind-steal-next pfind-queue-length pfind-parallelize-single-dir-access-using-hashing

However, nproc won't be supported at all, in #66, I wrote that spawning a bash from an MPI process that executes MPI doesn't necessarily work. With current OpenMPI it doesn't anymore. Therefore, needs a changed/new pfind using the same API.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_VI4IO_io-2D500-2Ddev_issues_70-23issuecomment-2D637039719&d=DwMCaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=9OKz58FWoBqvzbfOfhblug_f1x6fCy6IXlMMGcjo22s&m=H5qo_8_7FdDsg1NxR-8I7WOj6FR0gDHN1Dresl7cF-c&s=8zhn1nZKkkm0P4hUedD0W5nLOqDw3crRkfTNH28ZvqA&e=, or unsubscribehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AAPT2PXQ6AJRY7W7YZRCPYDRUPZZFANCNFSM4NQBJDLA&d=DwMCaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=9OKz58FWoBqvzbfOfhblug_f1x6fCy6IXlMMGcjo22s&m=H5qo_8_7FdDsg1NxR-8I7WOj6FR0gDHN1Dresl7cF-c&s=L8HlTgkta2zKSP7fAE7G555Ln1pTjN2EpjnW8bPhIsc&e=.

JulianKunkel commented 4 years ago

I tried to do this. Didn't work any more with current OpenMPI. I searched in google for popen and found some issues revolving around that only mpi spawn can be used.

John Bent notifications@github.com schrieb am Mo., 1. Juni 2020, 19:57:

Can't you write a wrapper to pfind that is called by just rank 0 and it itself spawns a new MPI job?

From: Julian Kunkel notifications@github.com Sent: Monday, June 1, 2020 12:48 PM To: VI4IO/io-500-dev io-500-dev@noreply.github.com Cc: Subscribed subscribed@noreply.github.com Subject: Re: [VI4IO/io-500-dev] find phase: bash version doesn't look at named parameters in .ini file (#70)

Basically, the external-extra-args would then support: pfind-steal-next pfind-queue-length pfind-parallelize-single-dir-access-using-hashing

However, nproc won't be supported at all, in #66, I wrote that spawning a bash from an MPI process that executes MPI doesn't necessarily work. With current OpenMPI it doesn't anymore. Therefore, needs a changed/new pfind using the same API.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub< https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_VI4IO_io-2D500-2Ddev_issues_70-23issuecomment-2D637039719&d=DwMCaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=9OKz58FWoBqvzbfOfhblug_f1x6fCy6IXlMMGcjo22s&m=H5qo_8_7FdDsg1NxR-8I7WOj6FR0gDHN1Dresl7cF-c&s=8zhn1nZKkkm0P4hUedD0W5nLOqDw3crRkfTNH28ZvqA&e=>, or unsubscribe< https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AAPT2PXQ6AJRY7W7YZRCPYDRUPZZFANCNFSM4NQBJDLA&d=DwMCaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=9OKz58FWoBqvzbfOfhblug_f1x6fCy6IXlMMGcjo22s&m=H5qo_8_7FdDsg1NxR-8I7WOj6FR0gDHN1Dresl7cF-c&s=L8HlTgkta2zKSP7fAE7G555Ln1pTjN2EpjnW8bPhIsc&e=

.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/VI4IO/io-500-dev/issues/70#issuecomment-637043750, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABGW5SRWYTWBHUAZ6XVNZQ3RUP22XANCNFSM4NQBJDLA .

carlilek commented 4 years ago

Putting the flags (aside from nproc) into the external-extra-args does appear to work as a workaround. What I did for nproc in prior seasons was to "replace" the pfind command with the full mpirun -ppn -np blah -blah -blah pfind. I don't think that would work in this case because the C app has the mpirun command baked in.

carlilek commented 4 years ago

Ah, so external-extra-args would then only apply to the bash script, while I would need to keep the matching C options in the ini file as well. That explains the find portion of the results I submitted yesterday and this morning.

--Ken

On Jun 1, 2020, at 2:36 PM, Carlile, Ken carlilek@janelia.hhmi.org<mailto:carlilek@janelia.hhmi.org> wrote:

Yes, I suspect that would be a functional temporary workaround. I'm not seeing exactly where the parameters would need to be translated for the bash version (I think the io500.sh calls io500-fixed.sh directly, so I guess it would need to be there?)

On Jun 1, 2020, at 2:35 PM, Julian Kunkel notifications@github.com<mailto:notifications@github.com> wrote:

Could you fill them into the "external-extra-args" section? This relates to #66https://urldefense.com/v3/__https://github.com/VI4IO/io-500-dev/issues/66__;!!Eh6p8Q!UasW6FyNSVH0TIb2AvqewXBMTqXLvMYyL6UyIEiU6PYboe-6KX8T9MShm8VTa-Xt9LzM$, an unofficial bugfix is available in the app master.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/VI4IO/io-500-dev/issues/70*issuecomment-637033112__;Iw!!Eh6p8Q!UasW6FyNSVH0TIb2AvqewXBMTqXLvMYyL6UyIEiU6PYboe-6KX8T9MShm8VTa7HVlsn7$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AC3AFHIVZWO56NTHI2YNONLRUPYFLANCNFSM4NQBJDLA__;!!Eh6p8Q!UasW6FyNSVH0TIb2AvqewXBMTqXLvMYyL6UyIEiU6PYboe-6KX8T9MShm8VTa-aMwZJb$.

carlilek commented 4 years ago

This appears to be an issue with the ISC20 version of io500-app (github.com/VI4IO/io-500-app) rather than -dev, as it's an issue with the pre-processing in the io500.sh of that package. I will close this issue here and open a new one over there.