Batch upload of topography data

[pastewka]

We need the possibility to upload batches of topographies. For example, line scans can be taken as multiple individual files (often many) and it is tedious to upload them separately.

The open question is how to assign metadata in such a batch upload. Initially, all topographies should probably have the same metadata but there should be a way to batch-edit metadata.

[mcrot]

Regarding the batch edit

Maybe we can solve this by implementing the following workflow for the user:

• user selects one or multiple topographies to edit/delete (maybe this can be integrated into the search results, here we have already such a selection)
• user presses button "batch-edit" or simply "edit"
• an algorithm collects which fields all these items have in common and have the same value, e.g. the same sizes, the same detrend_mode.
• a form is dynamically built from these fields
• after submitting the form, all selected items get the new values for those fields

@pastewka , does that sound useful for the users?

[mcrot]

Regarding the batch upload

Here the workflow could be like this:

• Step 1: User can select 1 or more files. If one file is selected, everything works like now. If multiple files are selected, each one is checked individually. All files must result in the same dimensionality, physical sizes etc: The channels of all files must have the same meta data.
• Step 2: Same as for one file, but all file must use the same datasource (I think the files will have the same datasources most of the time for a batch upload) Instead of a name, a prefix is entered by the user. The effective name will be built from the prefix and a counter or the file name.
• Step 3: Same as for one topography, the chosen sizes/units/scale/detrend_mode is used for all topographies

@pastewka , how does this sound?

[mcrot]

Another question: Should a user be able to create topographies from multiple datasources after uploading one file? This is not possible with the workflow described above. I think it is much more common to use the same one channel from many files.

[pastewka]

This sounds complicated. Would it make sense to have no form when uploading files, i.e. we prepopulate everything with default values? Batch upload would then be trivial.

The user then needs to edit topographies manually after upload. We probably should make this easier for the user, i.e. we could redirect to the edit form directly after upload. (This would be similar to what we have now, but without requiring to enter any information.)

I like the suggestion for selecting multiple topographies, but I would always show all field. Where field entries do not agree across the selection, fields should be gray and it should be clear that overwriting those entries will lead to all topographies having the same values. When batch uploading we redirect to the batch edit feature of all uploaded topographies.

This way we would need to implement two features:

• (Batch) upload
• (Batch) edit We could get rid of the special form shown after upload. (Is this a separate form in the code?) This would just be the edit form.

[mcrot]

I agree that it is a good idea to allow changing all fields across all topographies.

Regarding the upload: It would be nice to have only one edit form and not the multi-step process. Yes, these are extra forms for upload, metadata, and units.

This multi-step upload process has been used, because the steps depend on each other. So I need a data source to be chosen before I know whether physical sizes are given or not and for how many dimensions. I have to think about this and see how this matches the requirements we already have (e.g. not allowing to edit physical sizes any more if given in the file, and others).

Additionally, we should avoid to have any topographies in the system with default values which have not been edited yet (e.g. line scan with dimension 2??). Currently, this cannot happen, because a topography in the database is created only after all steps (upload, metadata, units) have been processed successfully.

Probably this is more complicated than I thought and we should discuss that in person.

Probably it's easier to discuss that in person.

[mcrot]

Result of the discussion today:

• replace the upload and edit of single topographies by a batch upload and a batch edit

• batch upload:

  • first page: several files can be given by file dialogue or by drag-n-drop
  • second page: name and data source are displayed in an table, user can change each of those, description+tags are not shown here
  • third page: physical sizes, unit, detrend_mode, .. all fields are displayed as table

  After confirming, all topographies are generated. Description + tags must be set through batch edit.

• batch edit: Topographies are selected in search view. After pressing edit topographies, all fields are shown once. Each field falls in one of these categories:
  • fields that have all the same values for all selected topographies: can be edited, shown white
  • fields that have different values among topographies: can be edited, shown in gray or another color, will be applied to all topographies, this is written in some help text as a warning
  • fields those can't be edited at all (like datasource, or name if not editing a single topography)

[pastewka]

@tdbjacobs requested to prioritize this

[pastewka]

After thinking about this a bit more, I think we need to restructure the way we interact with the topography information right now.

• Upload should require zero user interaction. This way we can simply batch upload many datasets. Topographies with unsupported file formats are rejected and the user is informed about this.
• Topographies with supported file formats can be incomplete if required information (e.g. the physical size of the scan) is missing. No analysis is run on incomplete datasets.
• The user can then at any later point edit the data set and provide that information. Analyses then run.

We would get rid of the wizard entirely. This has the advantage that it would significantly simplify the codebase, with only a single point remaining where data can be edited.

[tdbjacobs]

I like that plan because:

1. It facilitates batch upload
2. My students complain about the time and tedium of uploading 3 simpler is almost always in terms of encouraging use.

My only concern is that the user will upload the data and never edit it at all. This is ok, but eliminates some of the benefit of our tool (such as including reliability cut off where instrument/tip radius is put in)

(In more extreme cases, where a critical piece of info is missing (like scan size) we’ll need to throw a flag so the user is guided to enter the info)

A “nice to have” is as follows: If a user is uploading a batch of similar files (e.g. one hundred AFM images taken under the same conditions) it would be nice if there were a way to apply a modification to many datasets as once, e.g. “Set all of these as instrument: AFM” But if that’s not possible, it’s not a big deal.

Tevis

On Sat, Apr 1, 2023 at 3:26 AM Lars Pastewka @.***> wrote:

After thinking about this a bit more, I think we need to restructure the way we interact with the topography information right now.

• Upload should require zero user interaction. This way we can simply batch upload many datasets. Topographies with unsupported file formats are rejected and the user is informed about this.
• Topographies with supported file formats can be incomplete if required information (e.g. the physical size of the scan) is missing. No analysis is run on incomplete datasets.
• The user can then at any later point edit the data set and provide that information. Analyses then run.

We would get rid of the wizard entirely. This has the advantage that it would significantly simplify the codebase, with only a single point remaining where data can be edited.

— Reply to this email directly, view it on GitHub https://github.com/ContactEngineering/topobank/issues/173#issuecomment-1492862572, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFOUYWVCBXR4LORPP3A62SLW67KDFANCNFSM4HGUUTEA . You are receiving this because you were mentioned.Message ID: @.***>

-- Tevis Jacobs Mechanical Engineering and Materials Science University of Pittsburgh

[Sent from my phone. Please forgive typos.]

[pastewka]

Resolved with #1012