lvclark
commented
8 months ago (And I'm very experienced with other languages but not at all with Julia, so apologies if this is something obvious.)
Closed lvclark closed 8 months ago
lvclark
commented
8 months ago (And I'm very experienced with other languages but not at all with Julia, so apologies if this is something obvious.)
ranafi
commented
8 months ago This repository was created and is maintained by Jan Hammarlund (He programmedCYCLOPS V2) I will forward to him Thanks for your interest!
On Mon, Mar 4, 2024 at 3:30 PM Lindsay Clark @.***> wrote:
I have gotten this error on Julia 1.10 and 1.6 using Apptainer on a HPC cluster, as well as using Julia 1.10 on Windows. In the template script, when I run the line @everywhere include(path_to_cyclops) I get the error
ERROR: On worker 2: UndefVarError: path_to_cyclops not defined Stacktrace: [1] top-level scope @ none:1 [2] eval @ ./boot.jl:360 [3] #114 @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:293 [4] run_work_thunk @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:63 [5] run_work_thunk @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:72 [6] #100 @ ./task.jl:417
...and 4 more exceptions.
Stacktrace: [1] sync_end(c::Channel{Any}) @ Base ./task.jl:369 [2] macro expansion @ ./task.jl:388 [inlined] [3] remotecall_eval(m::Module, procs::Vector{Int64}, ex::Expr) @ Distributed /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/macros.jl:223 [4] top-level scope @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/macros.jl:207
So it seems that the additional worker processes don't have access to my same environment? I did make the path_to_cyclops variable, even made it right before running include just to be sure.
I have changed the preceding line to Distributed.addprocs(5) because I don't want to use all available cores, but I think I was seeing the error even before I made that change.
— Reply to this email directly, view it on GitHub https://github.com/ranafi/CYCLOPS-2.0/issues/1, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACNTTYAPSDBTWUTQAG6PY23YWTKXJAVCNFSM6AAAAABEF3ZGDSVHI2DSMVQWIX3LMV43ASLTON2WKOZSGE3DONRYHE3TSNY . You are receiving this because you are subscribed to this thread.Message ID: @.***>
ranafi
commented
8 months ago I will say that I do not think Jan ran this on a cluster (only a "powerdesktop" with a bunch of cores..)
On Mon, Mar 4, 2024 at 3:38 PM Ron Anafi @.***> wrote:
This repository was created and is maintained by Jan Hammarlund (He programmedCYCLOPS V2) I will forward to him Thanks for your interest!
On Mon, Mar 4, 2024 at 3:30 PM Lindsay Clark @.***> wrote:
I have gotten this error on Julia 1.10 and 1.6 using Apptainer on a HPC cluster, as well as using Julia 1.10 on Windows. In the template script, when I run the line @everywhere include(path_to_cyclops) I get the error
ERROR: On worker 2: UndefVarError: path_to_cyclops not defined Stacktrace: [1] top-level scope @ none:1 [2] eval @ ./boot.jl:360 [3] #114 @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:293 [4] run_work_thunk @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:63 [5] run_work_thunk @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:72 [6] #100 @ ./task.jl:417
...and 4 more exceptions.
Stacktrace: [1] sync_end(c::Channel{Any}) @ Base ./task.jl:369 [2] macro expansion @ ./task.jl:388 [inlined] [3] remotecall_eval(m::Module, procs::Vector{Int64}, ex::Expr) @ Distributed /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/macros.jl:223 [4] top-level scope @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/macros.jl:207
So it seems that the additional worker processes don't have access to my same environment? I did make the path_to_cyclops variable, even made it right before running include just to be sure.
I have changed the preceding line to Distributed.addprocs(5) because I don't want to use all available cores, but I think I was seeing the error even before I made that change.
— Reply to this email directly, view it on GitHub https://github.com/ranafi/CYCLOPS-2.0/issues/1, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACNTTYAPSDBTWUTQAG6PY23YWTKXJAVCNFSM6AAAAABEF3ZGDSVHI2DSMVQWIX3LMV43ASLTON2WKOZSGE3DONRYHE3TSNY . You are receiving this because you are subscribed to this thread.Message ID: @.***>
JanHammarlund
commented
8 months ago Hello Lindsay. The error appears to be related to either Apptainer or the HPC. The current parallelization method (using Distributed.jl and @ everywhere) doesn't really "play nice" with HPC architecture. As a lab, we are working on a cluster-compatible version of CYCLOPS 2.0. For now, could you please try running the script on a local machine and see if the error persists?
lvclark
commented
8 months ago Thanks for the quick response! It did happen on my Windows laptop as well using Julia 1.10.
JanHammarlund
commented
8 months ago Thank you for this insight. May I ask that you please provide the script file which you are trying to run so I may have a look?
lvclark
commented
8 months ago I just confirmed that I am getting the error. Here is the script. I also had to replace tab stops with spaces to keep it from printing function lists over and over again when pasted to the console.
using DataFrames, Statistics, StatsBase, LinearAlgebra, MultivariateStats, PyPlot, Distributed, Random, CSV, Revise, Distributions, Dates, MultipleTesting
#base_path = joinpath("/mnt") # define the base path in which the data file, the seed gene list, and the sample collection time file are located
base_path = joinpath(homedir(), "OneDrive - SCH", "Documents", "CIRAEC", "powell_2024-02_cyclops")
data_path = joinpath(base_path, "muppits_ureca", "processed_data") # path for data folder in base folder
path_to_cyclops = joinpath(base_path, "CYCLOPS.jl") # path to CYCLOPS.jl
output_path = joinpath(base_path, "muppits_ureca", "training_results") # real training run output folder in base folder
expression_data = CSV.read(joinpath(data_path, "full_data_covariates_RPM_2024-03-01.csv"), DataFrame) # load data file in data folder
seed_genes = ["BMAL1", "RORC", "CRY1", "NFIL3", "RBM3", "ZNF326", "SRSF3", "IFI44", "SFPQ", "CXCL1", "LAMA2", "SAT1", "TBC1D8", "PARP14", "CCT6B", "DYNLT4", "OXNAD1", "SLC25A35", "LINGO4", "CXCL8", "RHBDD2", "DDX60", "KHDC4", "MX2", "MX1", "REXO5", "GDAP1", "LMOD1", "SOX15", "SLC23A2", "GSTM4", "FER1L6", "SLC27A4", "WTIP", "HNRNPDL", "RPS6KL1", "HERC5", "PLXDC1", "PKIB", "ICAM2", "PRMT9", "PER2", "DUOX2", "CXCL3", "LRRC36", "SLC25A45", "VSIG10", "CCNB1", "ZNF589", "OVGP1", "GNMT", "TLCD3B", "POU6F1", "SH3D21", "SORCS2", "THEM5", "CHRNA9", "KIF23", "CMPK2", "SFTPD", "TMEM119", "PLK1", "BTAF1", "SLC16A11", "SLC15A1", "GPAT3", "CKS2", "RSAD2", "APOLD1", "ARG2", "ISG15", "PLB1", "CENPE", "BORA", "RAET1G", "PAPPA2", "CDC6", "FOS", "ODC1", "CCDC88B", "DOC2B", "SPATA6L", "HLA-DOA", "TSPYL2", "VGLL3", "IFI44L", "TNFAIP2", "DEFB4A", "HBEGF", "CNTNAP1", "CCDC15", "MYL5", "FKBP1B", "ZNF467", "HSPA1B", "EPSTI1", "GAS1", "CYP2R1", "ALPL", "PHF24"] # vector of strings containing gene symbols matching the format of gene symbols in the first column of the expression_data dataframe.
sample_ids_with_collection_times = [] # sample ids for which collection times exist
sample_collection_times = [] # colletion times for sample ids
if ((length(sample_ids_with_collection_times)+length(sample_collection_times))>0) && (length(sample_ids_with_collection_times) != length(sample_collection_times))
error("ATTENTION REQUIRED! Number of sample ids provided (\'sample_ids_with_collection_times\') must match number of collection times (\'sample_collection_times\').")
end
# make changes to training parameters, if required. Below are the defaults for the current version of cyclops.
training_parameters = Dict(:regex_cont => r".*_C", # What is the regex match for continuous covariates in the data file
:regex_disc => r".*_D", # What is the regex match for discontinuous covariates in the data file
:blunt_percent => 0.975, # What is the percentile cutoff below (lower) and above (upper) which values are capped
:seed_min_CV => 0.14, # The minimum coefficient of variation a gene of interest may have to be included in eigen gene transformation
:seed_max_CV => 0.9, # The maximum coefficient of a variation a gene of interest may have to be included in eigen gene transformation
:seed_mth_Gene => 10000, # The minimum mean a gene of interest may have to be included in eigen gene transformation
:norm_gene_level => true, # Does mean normalization occur at the seed gene level
:norm_disc => false, # Does batch mean normalization occur at the seed gene level
:norm_disc_cov => 1, # Which discontinuous covariate is used to mean normalize seed level data
:eigen_reg => true, # Does regression again a covariate occur at the eigen gene level
:eigen_reg_disc_cov => 1, # Which discontinous covariate is used for regression
:eigen_reg_exclude => false, # Are eigen genes with r squared greater than cutoff removed from final eigen data output
:eigen_reg_r_squared_cutoff => 0.6, # This cutoff is used to determine whether an eigen gene is excluded from final eigen data used for training
:eigen_reg_remove_correct => false, # Is the first eigen gene removed (true --> default) or it's contributed variance of the first eigne gene corrected by batch regression (false)
:eigen_first_var => false, # Is a captured variance cutoff on the first eigen gene used
:eigen_first_var_cutoff => 0.85, # Cutoff used on captured variance of first eigen gene
:eigen_total_var => 0.85, # Minimum amount of variance required to be captured by included dimensions of eigen gene data
:eigen_contr_var => 0.05, # Minimum amount of variance required to be captured by a single dimension of eigen gene data
:eigen_var_override => true, # Is the minimum amount of contributed variance ignored
:eigen_max => 5, # Maximum number of dimensions allowed to be kept in eigen gene data
:out_covariates => true, # Are covariates included in eigen gene data
:out_use_disc_cov => true, # Are discontinuous covariates included in eigen gene data
:out_all_disc_cov => true, # Are all discontinuous covariates included if included in eigen gene data
:out_disc_cov => [1,2], # Which discontinuous covariates are included at the bottom of the eigen gene data, if not all discontinuous covariates
:out_use_cont_cov => false, # Are continuous covariates included in eigen data
:out_all_cont_cov => true, # Are all continuous covariates included in eigen gene data
:out_use_norm_cont_cov => false, # Are continuous covariates Normalized
:out_all_norm_cont_cov => true, # Are all continuous covariates normalized
:out_cont_cov => 1, # Which continuous covariates are included at the bottom of the eigen gene data, if not all continuous covariates, or which continuous covariates are normalized if not all
:out_norm_cont_cov => 1, # Which continuous covariates are normalized if not all continuous covariates are included, and only specific ones are included
:init_scale_change => true, # Are scales changed
:init_scale_1 => false, # Are all scales initialized such that the model sees them all as having scale 1
# Or they'll be initilized halfway between 1 and their regression estimate.
:train_n_models => 80, # How many models are being trained
:train_μA => 0.001, # Learning rate of ADAM optimizer
:train_β => (0.9, 0.999), # β parameter for ADAM optimizer
:train_min_steps => 1500, # Minimum number of training steps per model
:train_max_steps => 2050, # Maximum number of training steps per model
:train_μA_scale_lim => 1000, # Factor used to divide learning rate to establish smallest the learning rate may shrink to
:train_circular => false, # Train symmetrically
:train_collection_times => false, # Train using known times
:train_collection_time_balance => 1.0, # How is the true time loss rescaled
# :train_sample_id => sample_ids_with_collection_times,
# :train_sample_phase => sample_collection_times,
:cosine_shift_iterations => 192, # How many different shifts are tried to find the ideal shift
:cosine_covariate_offset => true, # Are offsets calculated by covariates
:align_p_cutoff => 0.05, # When aligning the acrophases, what genes are included according to the specified p-cutoff
:align_base => "radians", # What is the base of the list (:align_acrophases or :align_phases)? "radians" or "hours"
:align_disc => false, # Is a discontinuous covariate used to align (true or false)
:align_disc_cov => 1, # Which discontinuous covariate is used to choose samples to separately align (is an integer)
:align_other_covariates => false, # Are other covariates included
:align_batch_only => false,
# :align_samples => sample_ids_with_collection_times,
# :align_phases => sample_collection_times,
# :align_genes => Array{String, 1}, # A string array of genes used to align CYCLOPS fit output. Goes together with :align_acrophases
# :align_acrophases => Array{<: Number, 1}, # A number array of acrophases for each gene used to align CYCLOPS fit output. Goes together with :align_genes
:X_Val_k => 10, # How many folds used in cross validation.
:X_Val_omit_size => 0.1, # What is the fraction of samples left out per fold
:plot_use_o_cov => true,
:plot_correct_batches => true,
:plot_disc => false,
:plot_disc_cov => 1,
:plot_separate => false,
:plot_color => ["b", "orange", "g", "r", "m", "y", "k"],
:plot_only_color => true,
:plot_p_cutoff => 0.05)
Distributed.addprocs(5)
@everywhere include(path_to_cyclops)
# real training run
training_parameters[:align_genes] = CYCLOPS.human_homologue_gene_symbol[CYCLOPS.human_homologue_gene_symbol .!= "RORC"]
training_parameters[:align_acrophases] = CYCLOPS.mouse_acrophases[CYCLOPS.human_homologue_gene_symbol .!= "RORC"]
eigendata, modeloutputs, correlations, bestmodel, parameters = CYCLOPS.Fit(expression_data, seed_genes, training_parameters)
CYCLOPS.Align(expression_data, modeloutputs, correlations, bestmodel, parameters, output_path)Also, I have one discontinuous covariate that is a batch effect, and one discontinuous covariate that is an effect of interest. I am a little confused about which should go into norm_disc_cov vs. eigen_reg_disc_cov vs. out_disc_cov vs. align_disc_cov.
And one last note, in CYCLOPS.jl I changed ARNTL to BMAL1 because of how my genes are annotated.
JanHammarlund
commented
8 months ago To answer your first issue:
# First run
Distributed.addprocs(5)
# then run
@everywhere path_to_cyclops = joinpath(base_path, "CYCLOPS.jl")
# finally run
@everywhere include(path_to_cyclops)Alternately:
# First run
Distributed.addprocs(5)
# then run
@everywhere include(joinpath(base_path, "CYCLOPS.jl"))Thank you! I also had to do
@everywhere base_path = joinpath(homedir(), "OneDrive - SCH", "Documents", "CIRAEC", "powell_2024-02_cyclops")to get it to work.
I have gotten this error on Julia 1.10 and 1.6 using Apptainer on a HPC cluster, as well as using Julia 1.10 on Windows. In the template script, when I run the line
@everywhere include(path_to_cyclops)I get the errorSo it seems that the additional worker processes don't have access to my same environment? I did make the
path_to_cyclopsvariable, even made it right before runningincludejust to be sure.I have changed the preceding line to
Distributed.addprocs(5)because I don't want to use all available cores, but I think I was seeing the error even before I made that change.