WHOI-PO-HPC

This repository will contain all relevant information related to using parallel and distributed data analysis on the Poseidon cluster at the Woods Hole Oceanographic Institution (WHOI). This repository will specifically go over Dask, xarray and requesting resources on Poseidon and your local machine. Below is a README that I asked ChatGPT to generate for me, we can change it later but I was lazy.

Parallel and Distributed Computing on the Poseidon Cluster at WHOI

Welcome to the repository for all relevant information related to using parallel and distributed computing on the Poseidon cluster at the Woods Hole Oceanographic Institution (WHOI). This repository will specifically cover the following topics:

• Using Dask for parallel computing
• Leveraging xarray for handling multi-dimensional arrays
• Requesting resources on the Poseidon cluster
• Setting up your local machine for compatibility with Poseidon

Table of Contents

• Introduction
• Getting Started
  • Prerequisites
  • Installation
• Using Dask
• Working with xarray
• Requesting Resources on Poseidon
• Setting Up Your Local Machine
• Contributing
• License
• Contact

Introduction

This repository contains comprehensive information on utilizing parallel and distributed computing resources on the Poseidon cluster at WHOI. The primary focus is on using Dask for parallel computing, leveraging xarray for managing multi-dimensional arrays, and efficiently requesting resources on both the Poseidon cluster and your local machine.

Getting Started

Prerequisites

Before you begin, ensure you have met the following requirements:

• You have a WHOI Poseidon cluster account.
• You have Python installed on your local machine.
• You have basic knowledge of parallel and distributed computing concepts.

Installation

1. Clone the repository:

   git clone https://github.com/anthony-meza/WHOI-PO-HPC.git

2. Navigate to the project directory:

   cd WHOI-PO-HPC

3. Install the required Python packages: Using pip

   pip install -r requirements.txt

   or conda (preferred method)

   conda env create -f environment.yml

Using Dask

Dask is a flexible parallel computing library for analytics. It helps scale the Python ecosystem (numpy, pandas, scikit-learn, etc.) and enables execution on multi-core machines and distributed clusters.

• Dask Documentation
• Examples and tutorials: examples/

Working with xarray

xarray is an open-source project and Python package that makes working with labeled multi-dimensional arrays simple, efficient, and fun!

• xarray Documentation
• Examples and tutorials: examples/

Requesting Resources on Poseidon

Learn how to request and manage computing resources on the Poseidon cluster effectively:

• Poseidon User Guide
• Examples of job scripts: scripts/

Setting Up Your Local Machine

To ensure your local machine can interact with Poseidon:

• Install the necessary SSH tools.
• Set up environment modules for consistency with the cluster environment.

Contributing

Contributions are welcome! Please read the contributing guide to get started.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Contact

For any questions or suggestions, please open an issue or contact the repository maintainer at your-email@whoi.edu.