search

JaneliaSciComp / multifish

EASI-FISH analysis pipeline for spatial transcriptomics
BSD 3-Clause "New" or "Revised" License
33 stars 14 forks source link

demo_tiny issue on AWS #26

Closed FangmingXie closed 1 year ago

FangmingXie commented 1 year ago

I was trying to run through the demo_tiny example on Amazon AWS S3, following the default parameters as specified in demo_tiny.json and this tutorial: https://janeliascicomp.github.io/multifish/tower/NextflowTowerAWS.html.

The process stuck during stitching, specifically it reports this error:

[29/18bd94] NOTE: Process `stitching:stitch:spark_cluster:wait_for_master (1)` failed -- Execution is retried (1)

What is also confusing is that the process wanted to request 14 workers (screenshot below), although I only requested 1 worker followingdemo_tiny.json.

Could you help me figure out what went wrong? Below I also attached the parameters and the full execution log. Thank you!

image

warp_spots_memory = 2 G
data_manifest = demo_tiny
airlocalize_xy_stride = 512
segmentation_memory = 2 G
lsf_opts =
registration_transform_memory = 2 G
ref_acq = LHA3_R3_tiny
driver_memory = 1g
segmentation_cpus = 8
ransac_cpus = 8
workers = 1
registration_xy_stride = 512
airlocalize_z_overlap = 32
worker_cores = 4
dapi_channel = c1
stitching_czi_pattern = _V%02d
airlocalize_z_stride = 128
acq_names = LHA3_R3_tiny,LHA3_R5_tiny
N E X T F L O W  ~  version 22.10.7
Pulling JaneliaSciComp/multifish ...
downloaded from https://github.com/JaneliaSciComp/multifish.git
Launching `https://github.com/JaneliaSciComp/multifish` [mad_agnesi] DSL2 - revision: a7dc221c26 [master]
===================================
EASI-FISH ANALYSIS PIPELINE
===================================
Pipeline parameters
-------------------
workDir                : /fusion/s3/easi-fish-test1/trials_FX/scratch
data_manifest          : demo_tiny
shared_work_dir        : /fusion/s3/easi-fish-test1/trials_FX
segmentation_model_dir : /fusion/s3/easi-fish-test1/trials_FX/inputs/model/starfinity
data_dir               : /fusion/s3/easi-fish-test1/trials_FX/inputs
output_dir             : /fusion/s3/easi-fish-test1/trials_FX/outputs
publish_dir            : /fusion/s3/easi-fish-test1/trials_FX/publish
acq_names              : [LHA3_R3_tiny, LHA3_R5_tiny]
ref_acq                : LHA3_R3_tiny
steps_to_skip          : []
Monitor the execution with Nextflow Tower using this URL: https://tower.nf/user/fmxie1993/watch/2rvaNLW028Ljxc
[d0/c5fa43] Submitted process > collect_merge_points:collect_merged_points_files (1)
[fb/8f9c43] Submitted process > download (1)
[18/6bb3d6] Submitted process > stitching:prepare_stitching_data (1)
[a5/0009cc] Submitted process > stitching:prepare_stitching_data (2)
[01/85404b] Submitted process > stitching:stitch:spark_cluster:prepare_spark_work_dir (1)
[ed/31dc97] Submitted process > stitching:stitch:spark_cluster:prepare_spark_work_dir (2)
[29/18bd94] Submitted process > stitching:stitch:spark_cluster:wait_for_master (1)
[75/53d3c1] Submitted process > stitching:stitch:spark_cluster:spark_master (1)
[ab/a180bb] Submitted process > stitching:stitch:spark_cluster:wait_for_master (2)
[51/f5e9bb] Submitted process > stitching:stitch:spark_cluster:spark_master (2)
[29/18bd94] NOTE: Process `stitching:stitch:spark_cluster:wait_for_master (1)` failed -- Execution is retried (1)
[ab/a180bb] NOTE: Process `stitching:stitch:spark_cluster:wait_for_master (2)` failed -- Execution is retried (1)
[8f/1c4ed4] Re-submitted process > stitching:stitch:spark_cluster:wait_for_master (1)
[07/6e8720] Re-submitted process > stitching:stitch:spark_cluster:wait_for_master (2)
[8f/1c4ed4] NOTE: Process `stitching:stitch:spark_cluster:wait_for_master (1)` failed -- Execution is retried (2)
[b1/506ee2] Re-submitted process > stitching:stitch:spark_cluster:wait_for_master (1)
[b1/506ee2] NOTE: Process `stitching:stitch:spark_cluster:wait_for_master (1)` failed -- Execution is retried (3)
[fc/385e22] Re-submitted process > stitching:stitch:spark_cluster:wait_for_master (1)
[fc/385e22] NOTE: Process `stitching:stitch:spark_cluster:wait_for_master (1)` failed -- Execution is retried (4)
[a3/f52085] Re-submitted process > stitching:stitch:spark_cluster:wait_for_master (1)
[a3/f52085] NOTE: Process `stitching:stitch:spark_cluster:wait_for_master (1)` failed -- Execution is retried (5)
[e8/e35e42] Re-submitted process > stitching:stitch:spark_cluster:wait_for_master (1)
[e8/e35e42] NOTE: Process `stitching:stitch:spark_cluster:wait_for_master (1)` failed -- Execution is retried (6)
[5c/59bcd7] Re-submitted process > stitching:stitch:spark_cluster:wait_for_master (1)
[5c/59bcd7] NOTE: Process `stitching:stitch:spark_cluster:wait_for_master (1)` failed -- Execution is retried (7)
[96/1d03af] Re-submitted process > stitching:stitch:spark_cluster:wait_for_master (1)
[96/1d03af] NOTE: Process `stitching:stitch:spark_cluster:wait_for_master (1)` failed -- Execution is retried (8)
[4d/2271d2] Re-submitted process > stitching:stitch:spark_cluster:wait_for_master (1)
[4d/2271d2] NOTE: Process `stitching:stitch:spark_cluster:wait_for_master (1)` failed -- Execution is retried (9)
[8c/38df25] Re-submitted process > stitching:stitch:spark_cluster:wait_for_master (1)
cgoina commented 1 year ago

Can you post all the parameters? I am just trying the job on tower.nf with a new envioronment I just created and the job is way past the point where yours got stuck. Here are my parameters:

warp_spots_memory = 2 G
data_manifest = demo_tiny
airlocalize_xy_stride = 512
segmentation_memory = 2 G
lsf_opts =
registration_transform_memory = 2 G
ref_acq = LHA3_R3_tiny
skip = segmentation,spot_extraction,warp_spots,measure_intensities,assign_spots
driver_memory = 1g
segmentation_cpus = 8
ransac_cpus = 8
workers = 1
registration_xy_stride = 512
airlocalize_z_overlap = 32
worker_cores = 4
dapi_channel = c1
stitching_czi_pattern = _V%02d
airlocalize_z_stride = 128
acq_names = LHA3_R3_tiny,LHA3_R5_tiny
env-map:
PATH = /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/google-cloud-sdk/bin
AWS_BATCH_JQ_NAME = TowerForge-3GmJI84ly7k7IuBcDcQtnk
JAVA_HOME = /usr/lib/jvm/java-17-amazon-corretto
AWS_EXECUTION_ENV = AWS_ECS_EC2
NXF_OUT_FILE = nf-5w1cgS1vjZTNqE.txt
ECS_CONTAINER_METADATA_URI_V4 = http://169.254.170.2/v4/7a75f73b-f989-4f3d-8cc7-9d5b428c0ef6
ECS_CONTAINER_METADATA_URI = http://169.254.170.2/v3/7a75f73b-f989-4f3d-8cc7-9d5b428c0ef6
NXF_UUID = 0ff4f51d-a7b6-459d-89c5-e4ce1b8981fe
NXF_TML_FILE = timeline-5w1cgS1vjZTNqE.html
LANG = C.UTF-8
NXF_HOME = /.nextflow
NXF_DEFAULT_DSL = 1
ECS_AGENT_URI = http://169.254.170.2/api/7a75f73b-f989-4f3d-8cc7-9d5b428c0ef6
NXF_ORG = nextflow-io
NXF_ANSI_LOG = false
NXF_PLUGINS_DEFAULT = nf-tower,nf-amazon,xpack-amzn
NXF_VER = 22.10.7
TOWER_LAUNCH_UMASK = 0
CAPSULE_CACHE_DIR = /.nextflow/capsule
NXF_SCM_FILE = https://api.tower.nf/ephemeral/13wMYj0apuW_YxALGvQ4Xw
NXF_JVM_ARGS = -XX:InitialRAMPercentage=40 -XX:MaxRAMPercentage=75
PWD = /
NXF_IGNORE_RESUME_HISTORY = true
AWS_BATCH_JOB_ID = 60fd7f69-5f88-4b63-ae25-f0d41cebc399
NXF_WORK = /fsx/work
AWS_BATCH_JOB_ATTEMPT = 1
NXF_CLI = /usr/local/bin/nextflow run https://github.com/JaneliaSciComp/multifish -name cheeky_mendel -params-file https://api.tower.nf/ephemeral/RTjaSI4KKXVoijLopPMcyw.json -with-tower
NXF_PACK = one
TOWER_WORKFLOW_ID = 5w1cgS1vjZTNqE
JAVA_CMD = /usr/lib/jvm/java-17-amazon-corretto/bin/java
NXF_XPACK_LICENSE = eyJ2ZXIiOjF9LnsiaWQiOiI0cUhvZ2d6ZktqaFNNb3FXRzJUTkdYIiwicHJvZCI6InhwYWNrLWdvb2dsZSx4cGFjay1hbXpuIiwiYWN0IjoiMjAyMS0wNy0yOVQxNToxOTo0MloiLCJleHAiOiIyMDIzLTExLTAxVDAwOjAwOjAwWiJ9LjExMDQ1N2RlMjMwNWEzYWI1YWRkZWQ5MGNlOTM4Mzc3OTEzYzY3Mzg=
HOSTNAME = ip-10-0-0-4.ec2.internal
NXF_PRERUN_BASE64 = ZXhwb3J0IFRPV0VSX0FDQ0VTU19UT0tFTj1leUpoYkdjaU9pSklVekkxTmlKOS5leUp6ZFdJaU9pSXhOell4SWl3aWJtSm1Jam94TmpjNU9UUXlOek0xTENKeWIyeGxjeUk2V3lKMWMyVnlJbDBzSW1semN5STZJblJ2ZDJWeUxXRndjQ0lzSW1WNGNDSTZNVFkzT1RrME5qTXpOU3dpYVdGMElqb3hOamM1T1RReU56TTFmUS5HMmVaMWlvX2FnNVhDWTJsSjNTNGRTeS12QlpfNEI5emVvMWt6dVFhM3c4CmV4cG9ydCBUT1dFUl9SRUZSRVNIX1RPS0VOPWV5SmhiR2NpT2lKSVV6STFOaUo5Lk5EQTRORFZtWVRFdE4yTXlNUzAwT1dVeUxUazRZMlF0TnpBeFpqRmtZV0ZpWVdVMC5RWVJEeXlwb2sySDVpUEE5SXRQc1dqRkRyUXBsMG5lc3h4emVXVktqX3AwCmV4cG9ydCBOWEZfU0NNX0ZJTEU9aHR0cHM6Ly9hcGkudG93ZXIubmYvZXBoZW1lcmFsLzEzd01ZajBhcHVXX1l4QUxHdlE0WHcKZXhwb3J0IE5YRl9YUEFDS19MSUNFTlNFPSdodHRwczovL2FwaS50b3dlci5uZi9lcGhlbWVyYWwvNmhIbEVsS1FlYTBqVXdwMUpQZE05UScK
AWS_BATCH_CE_NAME = TowerForge-3GmJI84ly7k7IuBcDcQtnk
NXF_ENABLE_SECRETS = true
NXF_FUSION_BUCKETS = s3://janelia-nextflow-demo
NXF_LOG_FILE = nf-5w1cgS1vjZTNqE.log
SHLVL = 1
HOME = /root
singularity_cache_dir = /root/.singularity_cache
registration_z_stride = 64
ransac_memory = 1 G
aff_scale = s1
shared_work_dir = /fsx/goinac/multifish-tiny
stitching_block_size = 1024,1024,256
retile_z_size = 128
aff_scale_transform_memory = 2 G
def_scale_transform_memory = 2 G
warp_spots_cpus = 4
envMap:
PATH = /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/google-cloud-sdk/bin
AWS_BATCH_JQ_NAME = TowerForge-3GmJI84ly7k7IuBcDcQtnk
JAVA_HOME = /usr/lib/jvm/java-17-amazon-corretto
AWS_EXECUTION_ENV = AWS_ECS_EC2
NXF_OUT_FILE = nf-5w1cgS1vjZTNqE.txt
ECS_CONTAINER_METADATA_URI_V4 = http://169.254.170.2/v4/7a75f73b-f989-4f3d-8cc7-9d5b428c0ef6
ECS_CONTAINER_METADATA_URI = http://169.254.170.2/v3/7a75f73b-f989-4f3d-8cc7-9d5b428c0ef6
NXF_UUID = 0ff4f51d-a7b6-459d-89c5-e4ce1b8981fe
NXF_TML_FILE = timeline-5w1cgS1vjZTNqE.html
LANG = C.UTF-8
NXF_HOME = /.nextflow
NXF_DEFAULT_DSL = 1
ECS_AGENT_URI = http://169.254.170.2/api/7a75f73b-f989-4f3d-8cc7-9d5b428c0ef6
NXF_ORG = nextflow-io
NXF_ANSI_LOG = false
NXF_PLUGINS_DEFAULT = nf-tower,nf-amazon,xpack-amzn
NXF_VER = 22.10.7
TOWER_LAUNCH_UMASK = 0
CAPSULE_CACHE_DIR = /.nextflow/capsule
NXF_SCM_FILE = https://api.tower.nf/ephemeral/13wMYj0apuW_YxALGvQ4Xw
NXF_JVM_ARGS = -XX:InitialRAMPercentage=40 -XX:MaxRAMPercentage=75
PWD = /
NXF_IGNORE_RESUME_HISTORY = true
AWS_BATCH_JOB_ID = 60fd7f69-5f88-4b63-ae25-f0d41cebc399
NXF_WORK = /fsx/work
AWS_BATCH_JOB_ATTEMPT = 1
NXF_CLI = /usr/local/bin/nextflow run https://github.com/JaneliaSciComp/multifish -name cheeky_mendel -params-file https://api.tower.nf/ephemeral/RTjaSI4KKXVoijLopPMcyw.json -with-tower
NXF_PACK = one
TOWER_WORKFLOW_ID = 5w1cgS1vjZTNqE
JAVA_CMD = /usr/lib/jvm/java-17-amazon-corretto/bin/java
NXF_XPACK_LICENSE = eyJ2ZXIiOjF9LnsiaWQiOiI0cUhvZ2d6ZktqaFNNb3FXRzJUTkdYIiwicHJvZCI6InhwYWNrLWdvb2dsZSx4cGFjay1hbXpuIiwiYWN0IjoiMjAyMS0wNy0yOVQxNToxOTo0MloiLCJleHAiOiIyMDIzLTExLTAxVDAwOjAwOjAwWiJ9LjExMDQ1N2RlMjMwNWEzYWI1YWRkZWQ5MGNlOTM4Mzc3OTEzYzY3Mzg=
HOSTNAME = ip-10-0-0-4.ec2.internal
NXF_PRERUN_BASE64 = ZXhwb3J0IFRPV0VSX0FDQ0VTU19UT0tFTj1leUpoYkdjaU9pSklVekkxTmlKOS5leUp6ZFdJaU9pSXhOell4SWl3aWJtSm1Jam94TmpjNU9UUXlOek0xTENKeWIyeGxjeUk2V3lKMWMyVnlJbDBzSW1semN5STZJblJ2ZDJWeUxXRndjQ0lzSW1WNGNDSTZNVFkzT1RrME5qTXpOU3dpYVdGMElqb3hOamM1T1RReU56TTFmUS5HMmVaMWlvX2FnNVhDWTJsSjNTNGRTeS12QlpfNEI5emVvMWt6dVFhM3c4CmV4cG9ydCBUT1dFUl9SRUZSRVNIX1RPS0VOPWV5SmhiR2NpT2lKSVV6STFOaUo5Lk5EQTRORFZtWVRFdE4yTXlNUzAwT1dVeUxUazRZMlF0TnpBeFpqRmtZV0ZpWVdVMC5RWVJEeXlwb2sySDVpUEE5SXRQc1dqRkRyUXBsMG5lc3h4emVXVktqX3AwCmV4cG9ydCBOWEZfU0NNX0ZJTEU9aHR0cHM6Ly9hcGkudG93ZXIubmYvZXBoZW1lcmFsLzEzd01ZajBhcHVXX1l4QUxHdlE0WHcKZXhwb3J0IE5YRl9YUEFDS19MSUNFTlNFPSdodHRwczovL2FwaS50b3dlci5uZi9lcGhlbWVyYWwvNmhIbEVsS1FlYTBqVXdwMUpQZE05UScK
AWS_BATCH_CE_NAME = TowerForge-3GmJI84ly7k7IuBcDcQtnk
NXF_ENABLE_SECRETS = true
NXF_FUSION_BUCKETS = s3://janelia-nextflow-demo
NXF_LOG_FILE = nf-5w1cgS1vjZTNqE.log
SHLVL = 1
HOME = /root
airlocalize_memory = 2 G
deform_memory = 2 G
channels = c0,c1
gb_per_core = 12
registration_stitch_memory = 2 G
airlocalize_cpus = 1
runtime_opts =
airlocalize_xy_overlap = 32
def_scale = s2
publish_dir = /fusion/s3/janelia-nextflow-demo/goinac-tiny-multifish
FangmingXie commented 1 year ago

Thanks for trying! Here are the full parameters. Anything wrong?

One thing I noticed is your parameters have skip = segmentation,spot_extraction,warp_spots,measure_intensities,assign_spots. Why skipping these? Did I miss something obvious? Thanks!

warp_spots_memory = 2 G
data_manifest = demo_tiny
airlocalize_xy_stride = 512
segmentation_memory = 2 G
lsf_opts =
registration_transform_memory = 2 G
ref_acq = LHA3_R3_tiny
driver_memory = 1g
segmentation_cpus = 8
ransac_cpus = 8
workers = 1
registration_xy_stride = 512
airlocalize_z_overlap = 32
worker_cores = 4
dapi_channel = c1
stitching_czi_pattern = _V%02d
airlocalize_z_stride = 128
acq_names = LHA3_R3_tiny,LHA3_R5_tiny
env-map:
PATH = /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/google-cloud-sdk/bin
AWS_BATCH_JQ_NAME = TowerForge-6W7tdqcFsDpwCrggOTbsKk
TOWER_ACCESS_TOKEN = eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiI4MTcxIiwibmJmIjoxNjc5ODkzNjAwLCJyb2xlcyI6WyJ1c2VyIl0sImlzcyI6InRvd2VyLWFwcCIsImV4cCI6MTY3OTg5NzIwMCwiaWF0IjoxNjc5ODkzNjAwfQ.Mq1yY7XJo2wWbRiKexdntnf6z42SN0w2U1IFaBiYVhw
JAVA_HOME = /usr/lib/jvm/java-17-amazon-corretto
AWS_EXECUTION_ENV = AWS_ECS_EC2
NXF_OUT_FILE = nf-2rvaNLW028Ljxc.txt
ECS_CONTAINER_METADATA_URI_V4 = http://169.254.170.2/v4/ecf55910-b252-4c1c-a9fb-e008a47c7175
ECS_CONTAINER_METADATA_URI = http://169.254.170.2/v3/ecf55910-b252-4c1c-a9fb-e008a47c7175
NXF_UUID = 0de1d78e-e8ed-4ba0-ad9f-298108b47294
NXF_TML_FILE = timeline-2rvaNLW028Ljxc.html
LANG = C.UTF-8
NXF_HOME = /.nextflow
NXF_DEFAULT_DSL = 1
ECS_AGENT_URI = http://169.254.170.2/api/ecf55910-b252-4c1c-a9fb-e008a47c7175
NXF_ORG = nextflow-io
NXF_ANSI_LOG = false
NXF_PLUGINS_DEFAULT = nf-tower,nf-amazon,xpack-amzn
NXF_VER = 22.10.7
CAPSULE_CACHE_DIR = /.nextflow/capsule
NXF_SCM_FILE = https://api.tower.nf/ephemeral/snuVu9ejQfSG2i3OfRZ5JA
NXF_JVM_ARGS = -XX:InitialRAMPercentage=40 -XX:MaxRAMPercentage=75
PWD = /
NXF_IGNORE_RESUME_HISTORY = true
AWS_BATCH_JOB_ID = a246e3c4-2b90-4484-95f8-e6a694d98b54
NXF_WORK = s3://easi-fish-test1/trials_FX/scratch
AWS_BATCH_JOB_ATTEMPT = 1
NXF_CLI = /usr/local/bin/nextflow run https://github.com/JaneliaSciComp/multifish -name mad_agnesi -params-file https://api.tower.nf/ephemeral/gvLV_86LLn1pGjoQIAJ37A.json -with-tower
NXF_PACK = one
TOWER_WORKFLOW_ID = 2rvaNLW028Ljxc
JAVA_CMD = /usr/lib/jvm/java-17-amazon-corretto/bin/java
NXF_XPACK_LICENSE = eyJ2ZXIiOjF9LnsiaWQiOiI0cUhvZ2d6ZktqaFNNb3FXRzJUTkdYIiwicHJvZCI6InhwYWNrLWdvb2dsZSx4cGFjay1hbXpuIiwiYWN0IjoiMjAyMS0wNy0yOVQxNToxOTo0MloiLCJleHAiOiIyMDIzLTExLTAxVDAwOjAwOjAwWiJ9LjExMDQ1N2RlMjMwNWEzYWI1YWRkZWQ5MGNlOTM4Mzc3OTEzYzY3Mzg=
TOWER_REFRESH_TOKEN = eyJhbGciOiJIUzI1NiJ9.NzU2YjE2ZmEtOWI0ZS00ZDdkLWEyNmQtNTk1ODUzMWQyZGE4.QEBh-8woDzdLfGjAhhjyXnRz-Rmq1nRbD6ocMfPlfrk
HOSTNAME = ip-172-31-6-164.us-west-1.compute.internal
NXF_PRERUN_BASE64 = ZXhwb3J0IFRPV0VSX0FDQ0VTU19UT0tFTj1leUpoYkdjaU9pSklVekkxTmlKOS5leUp6ZFdJaU9pSTRNVGN4SWl3aWJtSm1Jam94TmpjNU9Ea3pOakF3TENKeWIyeGxjeUk2V3lKMWMyVnlJbDBzSW1semN5STZJblJ2ZDJWeUxXRndjQ0lzSW1WNGNDSTZNVFkzT1RnNU56SXdNQ3dpYVdGMElqb3hOamM1T0Rrek5qQXdmUS5NcTF5WTdYSm8yd1diUmlLZXhkbnRuZjZ6NDJTTjB3MlUxSUZhQmlZVmh3CmV4cG9ydCBUT1dFUl9SRUZSRVNIX1RPS0VOPWV5SmhiR2NpT2lKSVV6STFOaUo5Lk56VTJZakUyWm1FdE9XSTBaUzAwWkRka0xXRXlObVF0TlRrMU9EVXpNV1F5WkdFNC5RRUJoLTh3b0R6ZExmR2pBaGhqeVhuUnotUm1xMW5SYkQ2b2NNZlBsZnJrCmV4cG9ydCBOWEZfU0NNX0ZJTEU9aHR0cHM6Ly9hcGkudG93ZXIubmYvZXBoZW1lcmFsL3NudVZ1OWVqUWZTRzJpM09mUlo1SkEKZXhwb3J0IE5YRl9YUEFDS19MSUNFTlNFPSdodHRwczovL2FwaS50b3dlci5uZi9lcGhlbWVyYWwvRlhNYjRLRUJ0X1VrV1hTemxLeWNQZycK
AWS_BATCH_CE_NAME = TowerForge-6W7tdqcFsDpwCrggOTbsKk
NXF_ENABLE_SECRETS = true
NXF_FUSION_BUCKETS = s3://janelia-nextflow-demo,s3://easi-fish-test1/trials_FX/
NXF_LOG_FILE = nf-2rvaNLW028Ljxc.log
SHLVL = 1
HOME = /root
singularity_cache_dir = /root/.singularity_cache
registration_z_stride = 64
ransac_memory = 1 G
aff_scale = s1
shared_work_dir = /fusion/s3/easi-fish-test1/trials_FX
stitching_block_size = 1024,1024,256
retile_z_size = 128
aff_scale_transform_memory = 2 G
def_scale_transform_memory = 2 G
warp_spots_cpus = 4
envMap:
PATH = /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/google-cloud-sdk/bin
AWS_BATCH_JQ_NAME = TowerForge-6W7tdqcFsDpwCrggOTbsKk
TOWER_ACCESS_TOKEN = eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiI4MTcxIiwibmJmIjoxNjc5ODkzNjAwLCJyb2xlcyI6WyJ1c2VyIl0sImlzcyI6InRvd2VyLWFwcCIsImV4cCI6MTY3OTg5NzIwMCwiaWF0IjoxNjc5ODkzNjAwfQ.Mq1yY7XJo2wWbRiKexdntnf6z42SN0w2U1IFaBiYVhw
JAVA_HOME = /usr/lib/jvm/java-17-amazon-corretto
AWS_EXECUTION_ENV = AWS_ECS_EC2
NXF_OUT_FILE = nf-2rvaNLW028Ljxc.txt
ECS_CONTAINER_METADATA_URI_V4 = http://169.254.170.2/v4/ecf55910-b252-4c1c-a9fb-e008a47c7175
ECS_CONTAINER_METADATA_URI = http://169.254.170.2/v3/ecf55910-b252-4c1c-a9fb-e008a47c7175
NXF_UUID = 0de1d78e-e8ed-4ba0-ad9f-298108b47294
NXF_TML_FILE = timeline-2rvaNLW028Ljxc.html
LANG = C.UTF-8
NXF_HOME = /.nextflow
NXF_DEFAULT_DSL = 1
ECS_AGENT_URI = http://169.254.170.2/api/ecf55910-b252-4c1c-a9fb-e008a47c7175
NXF_ORG = nextflow-io
NXF_ANSI_LOG = false
NXF_PLUGINS_DEFAULT = nf-tower,nf-amazon,xpack-amzn
NXF_VER = 22.10.7
CAPSULE_CACHE_DIR = /.nextflow/capsule
NXF_SCM_FILE = https://api.tower.nf/ephemeral/snuVu9ejQfSG2i3OfRZ5JA
NXF_JVM_ARGS = -XX:InitialRAMPercentage=40 -XX:MaxRAMPercentage=75
PWD = /
NXF_IGNORE_RESUME_HISTORY = true
AWS_BATCH_JOB_ID = a246e3c4-2b90-4484-95f8-e6a694d98b54
NXF_WORK = s3://easi-fish-test1/trials_FX/scratch
AWS_BATCH_JOB_ATTEMPT = 1
NXF_CLI = /usr/local/bin/nextflow run https://github.com/JaneliaSciComp/multifish -name mad_agnesi -params-file https://api.tower.nf/ephemeral/gvLV_86LLn1pGjoQIAJ37A.json -with-tower
NXF_PACK = one
TOWER_WORKFLOW_ID = 2rvaNLW028Ljxc
JAVA_CMD = /usr/lib/jvm/java-17-amazon-corretto/bin/java
NXF_XPACK_LICENSE = eyJ2ZXIiOjF9LnsiaWQiOiI0cUhvZ2d6ZktqaFNNb3FXRzJUTkdYIiwicHJvZCI6InhwYWNrLWdvb2dsZSx4cGFjay1hbXpuIiwiYWN0IjoiMjAyMS0wNy0yOVQxNToxOTo0MloiLCJleHAiOiIyMDIzLTExLTAxVDAwOjAwOjAwWiJ9LjExMDQ1N2RlMjMwNWEzYWI1YWRkZWQ5MGNlOTM4Mzc3OTEzYzY3Mzg=
TOWER_REFRESH_TOKEN = eyJhbGciOiJIUzI1NiJ9.NzU2YjE2ZmEtOWI0ZS00ZDdkLWEyNmQtNTk1ODUzMWQyZGE4.QEBh-8woDzdLfGjAhhjyXnRz-Rmq1nRbD6ocMfPlfrk
HOSTNAME = ip-172-31-6-164.us-west-1.compute.internal
NXF_PRERUN_BASE64 = ZXhwb3J0IFRPV0VSX0FDQ0VTU19UT0tFTj1leUpoYkdjaU9pSklVekkxTmlKOS5leUp6ZFdJaU9pSTRNVGN4SWl3aWJtSm1Jam94TmpjNU9Ea3pOakF3TENKeWIyeGxjeUk2V3lKMWMyVnlJbDBzSW1semN5STZJblJ2ZDJWeUxXRndjQ0lzSW1WNGNDSTZNVFkzT1RnNU56SXdNQ3dpYVdGMElqb3hOamM1T0Rrek5qQXdmUS5NcTF5WTdYSm8yd1diUmlLZXhkbnRuZjZ6NDJTTjB3MlUxSUZhQmlZVmh3CmV4cG9ydCBUT1dFUl9SRUZSRVNIX1RPS0VOPWV5SmhiR2NpT2lKSVV6STFOaUo5Lk56VTJZakUyWm1FdE9XSTBaUzAwWkRka0xXRXlObVF0TlRrMU9EVXpNV1F5WkdFNC5RRUJoLTh3b0R6ZExmR2pBaGhqeVhuUnotUm1xMW5SYkQ2b2NNZlBsZnJrCmV4cG9ydCBOWEZfU0NNX0ZJTEU9aHR0cHM6Ly9hcGkudG93ZXIubmYvZXBoZW1lcmFsL3NudVZ1OWVqUWZTRzJpM09mUlo1SkEKZXhwb3J0IE5YRl9YUEFDS19MSUNFTlNFPSdodHRwczovL2FwaS50b3dlci5uZi9lcGhlbWVyYWwvRlhNYjRLRUJ0X1VrV1hTemxLeWNQZycK
AWS_BATCH_CE_NAME = TowerForge-6W7tdqcFsDpwCrggOTbsKk
NXF_ENABLE_SECRETS = true
NXF_FUSION_BUCKETS = s3://janelia-nextflow-demo,s3://easi-fish-test1/trials_FX/
NXF_LOG_FILE = nf-2rvaNLW028Ljxc.log
SHLVL = 1
HOME = /root
airlocalize_memory = 2 G
deform_memory = 2 G
channels = c0,c1
gb_per_core = 12
registration_stitch_memory = 2 G
airlocalize_cpus = 1
runtime_opts =
airlocalize_xy_overlap = 32
def_scale = s2
publish_dir = /fusion/s3/easi-fish-test1/trials_FX/publish
FangmingXie commented 1 year ago

Now I understand this skip = segmentation,spot_extraction,warp_spots,measure_intensities,assign_spots. is probably an irrelevant difference as the problem happens at stitching, which is upstream of those.

One thing I may have forgot to mention: I used S3 throughout the pipeline without using FSx, otherwise I tried to follow the tutorial.

cgoina commented 1 year ago

Yes skip parameter is not relevant in this case as I just wanted stitching. I think using s3 for everything - including your work directory - explains why the stitching gets stuck. I remember that initially when I tried to get the pipeline to run on AWS and I used s3 for everything - the stitching would simply hang at some point. I don't remember whether it was at the same point but my recommendation is to use FSx for the work directory. Please use FSx and set the compute environment just like we recommend it and try again. Let us know if you still experience problems with FSx.

FangmingXie commented 1 year ago

Thanks Cristian, I appreciate your help! I will try FSx then.

As a separate issue, we still wanted to run everything using S3 for the benefit of low cost to run through our very large sample. Konrad had suggested us to do it this way. So I guess my question is: do you think this can be a quick fix (using S3 only), or we should just forget about it altogether?

krokicki commented 1 year ago

I thought we had tried it with S3 only, but maybe I'm misremembering.

If it's not working then we should try to get it working in the future @cgoina, because FSx can be expensive to keep idle.

FangmingXie commented 1 year ago

Thanks!