search

bigdataprocessor / bigdataprocessor2

BSD 2-Clause "Simplified" License
20 stars 4 forks source link

Hardware setup #144

Open tischi opened 3 years ago

tischi commented 3 years ago

@manerotoni @cikus3 @nornil

May I ask how the hardware setup for big image data storage, visualisation and analysis looks at your institutes? For example, are you storing the data on a central storage? What kind of computer are you using for data inspection (your laptop, a workstation, a virtual machine)? How is your computer connected to the data (LAN, WIFI)? How fast is the connection of the respective computer to the storage?

cikus3 commented 3 years ago

Briefly, at The Crick (> 100 research group) each group has approximately 5 TB of space (extendible, especially for people working with Light-sheet..); data are stored on big internal server that is backuped daily. We save the data acquired with the microscope on the microscope-PC, but right after the acquisition, we copy the data to the server (with 10 Gb/s network). Data inspection and processing is rarely done through the server, but after copying again the data to a processing-PC (I know, sound crazy this back & forth data movement..) because working through the server is not easy (connection not stable). There are at least 2 exceptions: 1) for the light-sheet, we do some processing (opposite fusion,..) on the microscope Luxendo PC 128 GB RAM; when PC is available),before copying the data to the server. 2) we have a dedicated high performance machine (only accessible remotely – 400 GB RAM, 4 GPU, 12x2 CPU) for big data analysis, directly connected to the server; the original plan was to have a very fast connection (Infiniband - “like reading data from the local hardisik”), but for tech issues this is still not available (unfortunately). For data processing/inspection we have some workstation with about 128 GB RAM, Nvidia Gtx 1080, 10 Gb connection; they are fine). I rarely use my laptop for data analysis, but instead I connect to the local workstations using the Windows remote connections. We are currently implementing OMERO and we also have a large Cluster for running complex/long data processing (but is not easy to access/need some training). Hope it helps, just tell me if you need more details. Also better ideas are welcome!