Open sageyu123 opened 5 months ago
surajit4444mondal
commented
4 months ago I ran the following command on pipeline.
python /common/python/suncasa-src/suncasa/eovsa/eovsa_synoptic_imaging_pipeline.py UDB20220614.ms --ncpu 1But the code hang at some point.
sageyu123
commented
4 months ago Environment?
Sijie Yu, PhD Associate Research Professor Department of Physics/Center for Solar-Terrestrial Research New Jersey Institute of Technology https://web.njit.edu/~sjyu/
On Wed, Apr 24, 2024 at 19:44 Surajit Mondal @.***> wrote:
I ran the following command on pipeline. python /common/python/suncasa-src/suncasa/eovsa/eovsa_synoptic_imaging_pipeline. py UDB20220614. ms --ncpu 1 But the code hang at some point. — Reply to this email directly, view it on GitHub, or
I ran the following command on pipeline.
python /common/python/suncasa-src/suncasa/eovsa/eovsa_synoptic_imaging_pipeline.py UDB20220614.ms --ncpu 1
But the code hang at some point.
— Reply to this email directly, view it on GitHub https://urldefense.com/v3/__https://github.com/suncasa/suncasa-src/issues/287*issuecomment-2075502186__;Iw!!DLa72PTfQgg!KRGbwSZEO1iTIFEUh6U5pTslh85Qdxx34NLdJ1RcdXLad78rd26G2kw4gwGNI6Oil7wL0ZJAfAa3yehGTW4CjD86hw$, or unsubscribe https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AFHPYC3CC4SGI7QA3WVV6JTY67VQJAVCNFSM6AAAAABGFSKHUKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZVGUYDEMJYGY__;!!DLa72PTfQgg!KRGbwSZEO1iTIFEUh6U5pTslh85Qdxx34NLdJ1RcdXLad78rd26G2kw4gwGNI6Oil7wL0ZJAfAa3yehGTW5rmYTRYQ$ . You are receiving this because you were assigned.Message ID: @.***>
surajit4444mondal
commented
4 months ago Ran loadpyenv3.8 on pipeline
EOVSA Synoptic Pipeline Update
The new EOVSA synoptic pipeline, version 2.0, is now available for testing. Below are the key details to help you get started:
How to Run the Pipeline
Command to execute the pipeline for a specific date (example for 2021 Nov 24):
Output Location: The output FITS files are stored at
/data1/eovsa/fits/synoptic/2021/11/24/. They are also accessible via the web at https://ovsa.njit.edu/fits/synoptic/2021/11/24/.File Naming Convention: Files follow a new naming convention to differentiate between types:
Note: The daily synoptic image aggregates all the 60-minute synoptic images from that day.
Important: Ensure you specify
--version v2.0when using the new code. Using the old version (--version v1.0) or omitting the version will default to the original pipeline.Local Testing: You can also test the pipeline locally using a UDBms file as input:
The results will be stored in your current directory.
Current Issues and Developments
Rotation Timing for 60-minute Files: The
eovsa.synoptic_60mimages currently align to 20 UT. We plan to adjust this to better reflect actual observation times.Multi-threading on Linux: The code is designed to leverage multiple CPU cores for parallel processing, which effectively splits the daily UDBms into multiple chunks to be processed independently across different cores. This feature works seamlessly on macOS. The runtime of one UBDms file is less than an hour. However, challenges arise when running the code on our pipeline server under Linux, where the process tends to hang. This issue may be related to how CASA handles file operations in a multiprocessing environment. As a temporary measure, please set the number of CPU threads to 1. The typical run time is 8 hours. I will explore this issue further and seek insights from how the ovro-lwa-solar pipeline handles multiprocessing, which could offer a solution. I took a peek at what you guys have implemented in the ovro code, and didn't find out why the EOVSA code failed.
Initial Imaging Model: The pipeline uses hourly image models for self-calibration, which may be unreliable during high flare activity or with poor instrumental calibration. A robust check is required to ensure a good initial model is used.