abuts
commented
4 years ago fixed mainly in Herbert. Only tests are here.
Closed abuts closed 4 years ago
abuts
commented
4 years ago fixed mainly in Herbert. Only tests are here.
Dear Alex,
I believe the parallel running of gen_sqw with @transform_sqw is broken, at least for my account, but it may also be broken for everyone.
I have tested gen_sqw with @transform_sqw with a Horace internal function (so no issues with paths etc.)
gen_sqw(spefile, par_file, sym_name(sqw_file,n), efix, emode, alatt, angdeg,...
u, v, psi, omega, dpsi, gl, gs,[50,50,50,50],urange_in,'transform_sqw',@(x)(symmetrise_sqw(x,[1/2 -1 0],[0 0 1],[0 0 0])),'tmp_only',true);%make the sqw file
and the result is the same as when we use our own symmetrisation routine, the code works correctly in sequential mode but does not work in parallel mode, see below Figure 1 (sequential running) shows all pixels for y<0 are mapped for y>0, whereas figure 2 (parallel running) has performed no symmetrisation. I tested this for both isiscompute and IDAaaS, same behaviour. It looks like the @transform_sqw option is simply ignored under parallel running, so this needs to be fixed first, then I think our code will probably run ok. This functionality was broken recently as it worked for us a few weeks ago.
Can you run the code below from your account and see if you get Figure 1 (correctly symmetrised) or Figure 2 (not symmetrised), i.e. is this a problem with my account or is it a fundamental issue with parallel operation that is affecting everyone.
Radu
cd /instrument/MERLIN/RBNumber/RB1820500/
addpath('/instrument/LET/RBNumber/RB1810670/mslice');
addpath('/instrument/LET/RBNumber/RB1810670/mslice/horace_addons');
MakeSQWfile_CoTiO3_18meV_feb19(7,[],'_test',[1 250]);
ext='_feb19_test_sym7';[s,s1]=cut_sqw2(sprintf('%s%s%s','/instrument/MERLIN/RBNumber/RB1820500/CoTiO3_18meV',ext,'.sqw'),struct('u',[0.5,-1,0],'v',[1,0,0],'uoffset',[0,0,0,0],'type','ppr'),[-3.3,0.0075,3.3],[-1,0.0075,3.3],[-5,5],[8.25-5,8.75+5]); cax=[10 600];plot_slice(smooth_slice(s,1),cax(1),cax(2),'flat',ms_get_colormap(1),'linear');brillouin_zones_tr(1);drawnow;
From: Radu Coldea Sent: 11 July 2020 14:32 To: Alex Buts Alex.Buts@stfc.ac.uk Subject: RE: problem running gen_sqw with @transform_sqw for symmetrisation
Dear Alex,
I have already tried the option 3 to run the code serially in the most up to date version on isiscompute, it took 4.2 hours and the final sqw file is correctly symmetrised, as far as I can tell in sequential operation gen_sqw works correctly. It is the parallel part that does not work. What is surprising is that no errors come up when running in the most up to date version, the symmetrisation bit is simply ignored.
I suspect as you inferred the issue is with the workers/paths, something like this. In the gen_sqw code the @transform_sqw refers to function symmetrisation_routine which is located in the experimental folder /instrument/MERLIN/RBNumber/RB1820500, so when I start the MakeSQW job from this folder this function is there, my question is when individual jobs are passed to workers in which folders do those processes start, the same folder as the parent or different, will the function symmetrisation_routine be known to them?
When NoMachine access was stopped a few months ago and we ported everything to be able to work on IDAaaS, we edited symmetrisation_routine to add the correct paths
addpath('/instrument/LET/RBNumber/RB1810670/mslice');
addpath('/instrument/LET/RBNumber/RB1810670/mslice/horace_addons');
addpath('/instrument/MERLIN/RBNumber/RB1820500/');
and the parallel code then worked fine on both isiscompute and IDAaaS with the “default” workers, as you advised in the past I did not want to make any changes to the default worker files, but make any changes require in our own code. Then we tested everything on both IDAaaS and isiscompute and packaging/symmerisation all worked fine, the same worked fine a few weeks go, but when I tried it yesterday the symmetrisation part did not work in parallel mode.
I suspect the issue with parallel processing using gen_sqw with custom transform_sqw functions is not just related to us but may be a more general issue that could be affecting others as well. If possible it would be best to try and solve this using the default worker files, i.e. without interfering with those. Can you run gen_sqw in parallel mode ok, could you test the @transform_sqw option with any custom operation that is in a function that is not part of the default Horace package, it could be anything, for example putting all intensities to be a constant. Can you get that simple case working for one single nxspe file or two packaged nxspe files in your account, keep that file first in your default starting folder and see if that works, then move to a different folder and get things working there or identify why is not working, then if this simple packaging works for you we try to replicate what you did in our case, if it does not work for you then it is a deeper problem that will affect everyone who wants to use custom functions.
Radu
From: Alex Buts Alex.Buts@stfc.ac.uk Sent: 11 July 2020 13:00 To: Radu Coldea radu.coldea@physics.ox.ac.uk Subject: Re: problem running gen_sqw with @transform_sqw for symmetrisation
Dear Radu,
That's the issue with parallel jobs. My main task during last 3 month is to address this issue. Can not say I have fully resolved the issue but achieved significant progress on the way.
It is fully separate from symmetization, as parallel just gives boost in performance.
The changes to file system, seems broken old parallelization - that's known, but I do not remember what works in old Horace and what does not. We need to investigate.
Three steps to to check that:
1) I seems made typo yesterday enabling old Horace for you. I remember you using you own worker_v1, which enables your custom code - -- this one needs to be modified.
Modify your worker_v1 to work as the standard one and try to run your job again. It the job have not reported changes in status in 5 min (or time to symetrize single nxspe file) its broken.
2) changing parallel framework -- may work.
After your worker is modified
try
pc = parallel_framework.
pc.parallel_framework = 'parpool'
Start gen_sqw and If the job is not changing its status in 5 min -- its dead. The fixture is in the new code incompatible with old Horace.
3) run symetrization code serially. This would allow you to see if old Horace produces correct symmetrization and new one -- does not. Have no idea what have changed there -- nothing should.
But at least I will know that there is the problem in Horace to address.
Regards,
Alex
On 11/07/2020 09:31, Radu Coldea wrote: