tomeichlersmith
commented
2 years ago Meeting Notes
May 16, 2022
Attendees:
- Chad, Tom (U of Minnesota)
- Andrew Melo (US-CMS Tier 3 rep)
- ~Carl Lundstedt (U of Nebraska Lincoln)~
- missed but good resource
- Douglas Johnson (U of Colorado Boulder)
Chad Intro
- Running OSG 3.3 on SL6 (mostly)
- R410 and C6100 machines
- 3 disks in HDFS per machine roughly
- Hopefully performance from NFS is close enough
- Will wash hands of it if HDFS is a requirement
- Chad works for CSE-IT in Research Support
- Op-Team washed hands of old machines
- Low-level cluster admin is too much for single grad student
Focus on simple solution that is long term supportable.
What can CSE-IT provide?
- OS, Auth, Home Dirs (with managed machines), attach to current NFS storage
Get additional storage from CSE-IT like home directories?
- Could get limited space in O(10TB), limited by 10Gbps link to different building.
- NFS was not limiting factor in our current tests
- No new drives coming, outsourcing data might be best option
Usable size of Hadoop?
- ~300TB with 3x replication
- Relatively speaking, this is not expensive
Disposable compute nodes?
As hardware dies we can get rid of it.
In number of cores, the cluster is comparable to Europe Tier 2 sites. Storage capacity is the biggest issue - Hadoop is dying without new hardware.
First Order
- Update to newer software OSG 3.3 EoL was many years ago
- Newer OS
- Won't be able to get data via gridftp soon
Workflow Control
- Can't control job of outside grid users
- Colorado uses NFS for long term storage and Hadoop for helping run grid jobs
Hardware Rec
Upgraded head-node would be very helpful.
Storage Element Only
Allows remote jobs to write to our area.
Needs
- Frontier squid cache
- CVMFS, singularity
- Hosted CE only if accepting grid jobs from outside
- Submit crab jobs via Hosted CE?
- Just need some certificates not a full CE
- xrootd server for storage element to transfer in/out of our site
- condor head node and submit node (could be the same, some think it is a security issue)
- login node should be same as submit node
- Foreman and puppet for provisioning
- Carl from UoN should be asked
Questions
- Resource intensive-ness of xrootd and head node?
- Bottleneck if in different building?
Chad's Reading of Docs
The first diagram on this page is a good top-level view. https://opensciencegrid.org/docs/compute-element/hosted-ce/
I'm looking over the Hosted CE requirements. Please review these:
- Not greater than 10,000 concurrently running OSG resource requests
- An existing compute cluster running on a supported operating system with a supported batch system: Grid Engine, HTCondor, LSF, PBS Pro/Torque, or Slurm. - I think this means we need worker nodes
- Outbound network connectivity from your cluster's worker nodes - Not a problem
- Temporary scratch space on each worker node - /export/scratch
- Allow SSH access to your local cluster's login host from a known IP address - Need details on what the login node provides
- Shared home directories on each cluster node - Not a problem with Managed Machines
- One or more Unix accounts on your cluster's submit server with the following capabilities:
- Accessible via SSH key - SSH keys are allowed
- Use of SSH remote port forwarding (AllowTcpForwarding yes) and SSH multiplexing (MaxSessions 10 or greater) - Not an issue
- Permission to submit jobs to your local cluster.
Other pieces OSG Repository available to all nodes - Easy to set with Puppet EPEL Repository for Singularity - Already set in Puppet CVMFS - Already set in Puppet It looks like the worker node containers are not to configure the worker nodes but to send jobs to the queue as a container.
Reach out to Bryan Lim at UW to talk about Condor and OSG.