Directory loading is still slow

[keflavich]

Describe the bug Loading directories with 100's of files - of any type - takes a long time.

To Reproduce Open a folder with >100 .images or .fits files.

Expected behavior An immediate list, e.g., what ls returns.

Screenshots or videos If applicable, add screenshots/videos to help explain your problem.

https://github.com/CARTAvis/carta/assets/143715/9508e6d7-96e8-41f3-8c02-4ce224b26b05

Platform info (please complete the following information):

• OS: Mac OS frontend, Redhat 8 backend
• Browser chrome
• Version 4.1

[e-koch]

Similar issues occur when attempting to load region files.

[keflavich]

To provide some context here: John Hibbard was leading a CARTA discussion session at the SSMID workshop (https://science.nrao.edu/facilities/alma/naasc-workshops/ssmid/index) and everyone in the room raised their hands when we asked "is directory loading (still) slow?"

[kswang1029]

To achieve effectively an "ls", in the preferences dialog (global tab -> file list), you can switch to that mode of directory context parsing. Have you tried that? By default, everything in the directory is scanned for supported image types. If your backend host is HDD-based, it may take time if there are many files and subdirectories (especially for casa images which are "directory"). Please let us know if switching the parsing mode helps or not. Thanks.

[keflavich]

Yes, this is already with

[kswang1029]

So is the backend HDD-based? Are there multiple users accessing the same server (with HDD storage) running multiple backends? Please share with us more details for diagnostics.

[keflavich]

I can only speak for my own system, but afaik it's at least partly SSD based on a lustre system. The HPC group doesn't reveal the full details, in general: https://help.rc.ufl.edu/doc/Storage.

Because we've had this conversation before, I had assumed it was something unique to our supercomputer (hipergator). But since everyone in the room has experienced this same problem, it's safe to assume that it's not unique to our filesystem.

[e-koch]

I have similar issues on an SSD on a MacBook Pro (2020) running OSX 12.6

[kswang1029]

Is everyone in the room accessing the same file system? Is that a tutorial event with a computing environment configured for hands-on? Or they are accessing different file systems (local laptop, remote server with ssd, remote server with HDD-based Lustre etc.)?

We had tested before with a usb 2.5" HDD with such use cases but did not see such performance (with cache cleared for sure). I would like to at least figure out the right setup for further investigation. Being able to reproduce the issue is the first step.

[kswang1029]

I have similar issues on an SSD on a MacBook Pro (2020) running OSX 12.6

Would you mind sending us a text file about what you see with a ls?

[keflavich]

These are all independent users operating on different filesystems. We weren't testing at the time, this was just a general report of this being a common pain point for users.

Unfortunately, I cannot provide a MWE, but I hope Eric and others can!

[kswang1029]

Ok. Thanks for the info. I will have a look.

[kswang1029]

@keflavich BTW, are there other pain points?

[keflavich]

@kswang1029 I have a few other crashes here and there that are hard to reproduce; I'll try to screengrab some and post new issues for them.

[kswang1029]

@kswang1029 I have a few other crashes here and there that are hard to reproduce; I'll try to screengrab some and post new issues for them.

Thank you! We might have known some of them but please share with us your collection! 🙂

[Kitchi]

To add to this, we also see relatively slow loading of files in the browser on the ilifu cluster running on a CephFS filesystem running on an SSD and CARTA 4.1.0.

[kswang1029]

To add to this, we also see relatively slow loading of files in the browser on the ilifu cluster running on a CephFS filesystem running on an SSD and CARTA 4.1.0.

Thanks for the info. We will perform detailed performance benchmarks