Service Account PizDaint - Multiple file upload

[antonelepfl]

The Unicore API allows this so I guess that we would need to implement it as well.

[rcsm17]

Could you provide me a real example of a multi file uploading request ?

[antonelepfl]

To run some Simulations of brain area, I need 2 files: 1) The configuration of the simulations 2) The script where I load modules, run some extra script steps and then the run simulation.

The same thing happens with Analysis, I usually specify: 1) Use Unicore Imports to get the simulation results 2) Analysis configuration file 3) Script that runs the job.

[antonelepfl]

If you were looking for some code of how to do it, it is something like

await Promise.all(inputs.map(input => uploadData(input, `${workingDirectory}/files`)));

function uploadData(dataToUpload, uploadURL) {
  const data = dataToUpload.Data;
  const target = dataToUpload.To;
  return axiosInstance({
    url: `${uploadURL}/${target}`,
    method: 'put',
    data,
  });
}

Or Py-Unicore code

[rcsm17]

I need a lot of time to implement this function. Can you run these kind of jobs whit a single zipping all yours file into one zip ?

[rcsm17]

@antonelepfl can you give me a complete example script where you submit a job with multiple files uploading ? It would be better a jupiter notebook that I can debug it easely.

[antonelepfl]

Hi @rcsm17 I have created a Jupyter notebook From there you would be able to download the notebook and test it locally changing the endpoint as you suggested.

[rcsm17]

@antonelepfl I've created the new endpoint for advanced user. Now you can work in pizdaint using the service account as a unicore system, so you don't need to change your code but only the url of the system, from 'https://brissago.cscs.ch:8080/DAINT-CSCS/rest/core' to 'https://bspsa.cineca.it/advanced/pizdaint/rest/core/'.

I tested your notebook and I think it works correctly.

Test the new endpoint and please let me know if you find out any bugs.

[antonelepfl]

Wow thank you very much for the hard work on this!

I'll test it more deeply on Monday and I'll post the issues.

[antonelepfl]

Hi @rcsm17 Can we please add a response header when I POST to /jobs so I am able to retrieve the job's header location.

If I see the response from Jureca for instance

And if I look at the headers that you return

I see that the Accesas-Control-Expose-Headers is not present and that is why I'm not able to get the job Location(job id) header. Apparently the browser needs this 'Accesas-Control-Expose-Headers' to be explicit so it knows what it is able to retrieve.

If you have some time can we try to add this header please?

IIRC this specific header is only needed for the POST on /jobs

[rcsm17]

Hi @antonelepfl I added the Access-Control-Expose-Headers headers field for Location and the other all default headers listed here. Check it out if it works.

[antonelepfl]

Perfect! I was able to launch the simulations.. They are in the queue. I'll close this one and create a new one for the rest of the issues I'm getting.

[antonelepfl]

Hi @rcsm17 I see that the Location header disappeared. Can you make it visible again?

[rcsm17]

Hi @antonelepfl it will be available soon

[rcsm17]

@antonelepfl any news ?

[antonelepfl]

Yeah that was working I was waiting for your confirmation :)