packing/unpacking just the organizational structure, not the files

[dsudar]

Hi Erick,

I got the question whether omero-cli-transfer could just pack up the organizational structure (Projects, Datasets, etc.) empty and then re-populate a different server with that same structure. The purpose is to populate the 2nd server using ln_s imports into that same structure while the original server had regular imports. And they want to do the ln_s imports from the original source location.

I saw the "--binaries none" option but that doesn't seem to do what I mean unless I'm missing something?

Thanks for a great tool!

• Damir

[erickmartins]

Hi Damir,

can you explain in more detail why exactly --binaries none doesn't work for this use case? It should generate just the XML file that can be unpacked with --ln_s.

[dsudar]

I'm probably just missing one small thing: I did indeed do the pack with --binaries none and that created the .xml in a directory. But then unpack doesn't have a .zip or .tar file to work with. And providing the .xml gives: ValueError: File is not a zip or tar file

[erickmartins]

you can point to the folder where the xml is with the --folder argument - that should do it!

[dsudar]

Ah yes thanks, that gets it very close. It created the Project and the Datasets but didn't put the Datasets into the Project.

While it runs, it fails on the import of every image (understandably) but it continues happily. However at the end it terminates during the linking phase with:

Matching source and destination images...
Creating and linking OMERO objects...
Image corresponding to Image:151 not found. Skipping.
Image corresponding to Image:152 not found. Skipping.
Image corresponding to Image:156 not found. Skipping.
Image corresponding to Image:155 not found. Skipping.
Traceback (most recent call last):
  File "/home/dsudar/miniforge3/envs/transfer/bin/omero", line 8, in <module>
    sys.exit(main())
  File "/home/dsudar/miniforge3/envs/transfer/lib/python3.8/site-packages/omero/main.py", line 126, in main
    rv = omero.cli.argv()
  File "/home/dsudar/miniforge3/envs/transfer/lib/python3.8/site-packages/omero/cli.py", line 1787, in argv
    cli.invoke(args[1:])
  File "/home/dsudar/miniforge3/envs/transfer/lib/python3.8/site-packages/omero/cli.py", line 1225, in invoke
    stop = self.onecmd(line, previous_args)
  File "/home/dsudar/miniforge3/envs/transfer/lib/python3.8/site-packages/omero/cli.py", line 1302, in onecmd
    self.execute(line, previous_args)
  File "/home/dsudar/miniforge3/envs/transfer/lib/python3.8/site-packages/omero/cli.py", line 1384, in execute
    args.func(args)
  File "/home/dsudar/miniforge3/envs/transfer/lib/python3.8/site-packages/omero_cli_transfer.py", line 186, in _wrapper
    return func(self, *args, **kwargs)
  File "/home/dsudar/miniforge3/envs/transfer/lib/python3.8/site-packages/omero_cli_transfer.py", line 291, in unpack
    self.__unpack(args)
  File "/home/dsudar/miniforge3/envs/transfer/lib/python3.8/site-packages/omero_cli_transfer.py", line 544, in __unpack
    populate_omero(ome, img_map, self.gateway,
  File "/home/dsudar/miniforge3/envs/transfer/lib/python3.8/site-packages/generate_omero_objects.py", line 683, in populate_omero
    create_rois(ome.rois, ome.images, img_map, conn)
  File "/home/dsudar/miniforge3/envs/transfer/lib/python3.8/site-packages/generate_omero_objects.py", line 521, in create_rois
    img_id_dest = img_map[img.id]
KeyError: 'Image:151'

So the linking of the Datasets into the Project would probably be next but never gets to that point.

[erickmartins]

That is intended behavior - unpack expects imports to actually succeed and will fail if they don't. (the many ways an import can fail make it hard to catch it, so at the moment it does not fail gracefully.) unpack expects the paths in your XML to still succeed as import targets.

If what you want is just replicating projects and datasets between servers, I feel like you'd be better served by just writing a little bit of Python (maybe with ezomero?) to loop over them - it's not a use case we support right now.

[dsudar]

Yes, that makes sense. However, I found that the pack/unpack workflow as you designed it DOES work fine for my purpose IFF the Projects/Datasets on the origin server are empty of any Images. So since that entire set of data will be removed from the origin server anyway, I can simply delete all the images first, then pack the empty structure and unpack on the new server and then populate the new server using inplace import. So the tool will actually serve my purpose with some fiddling. Thanks! Fine to close this issue.