search

leap-stc / data-management

Collection of code to manually populate the persistent cloud bucket with data
https://catalog.leap.columbia.edu/
Apache License 2.0
0 stars 6 forks source link

Large eddy simulations of atmospheric boundary layer (TurbuABL) #118

Open Sshamekh opened 6 months ago

Sshamekh commented 6 months ago

Dataset Name

ABL-LES

Dataset URL

No response

Description

The ABL-LES dataset simulates the atmospheric boundary layer across a wide range of turbulent regimes, from channel flow to strong convective cases, with various boundary conditions (e.g., surface heating, geostrophic wind, free tropospheric lapse rate). The data also includes several implemented scalars (e.g., chemicals with various sources). This dataset is ideal for studying turbulence and transport under a wide range of conditions, making it great for data-driven approaches, specifically for developing parameterizations of the atmospheric boundary layer that can be implemented in climate or weather models.

Size

10-15 TB

License

Unknown

Data Format

NetCDF

Data Format (other)

No response

Access protocol

Globus

Source File Organization

Data are organized by the simulations (set of boundary conditions), and further diveded to snapshots, and statistics.

Example URLs

No response

Authorization

Globus login

Transformation / Processing

Data does not require cleaning but it requires to be better organized

Target Format

Zarr

Comments

No response

jbusecke commented 6 months ago

Hey @Sshamekh, thanks for submitting this. Where is the data accessible?

Sshamekh commented 6 months ago

Hi Julius, They are on NCAR HPC

jbusecke commented 6 months ago

Ah ok that makes things (possibly) harder. Can you ask if there is a way for them to make them accessible via http or ftp in some way?

Sshamekh commented 6 months ago

Thanks Julius. I will reach out to the NCAR helpdesk to inquire about HTTP or FTP access. However, I wanted to check if using Globus would also be an option beforehand.

jbusecke commented 6 months ago

Possibly. Could you check if they subscribe to the Globus S3 Connector? If that is the case we might be able to move the data to an OSN bucket.

jbusecke commented 6 months ago

Please feel free to cc me in emails if you think that would be helpful.

jbusecke commented 2 months ago

Actually did you ever get an answer to this? If they have their own setup, that might make things easier.

Sshamekh commented 2 months ago

Here is their Globus : https://ncar-hpc-docs.readthedocs.io/en/latest/storage-systems/data-transfer/globus/ But they didn't mention S3 connector. I will check with them

jbusecke commented 1 month ago

Hi @Sshamekh, just wanted to check in on the status of the transfer? Do you have some time to let it run some time soon?