struct.error: 'i' format requires -2147483648 <= number <= 2147483647

[Allen188]

Have an error while use code ：pyprophet score --in=model.osw --classifier=XGBoost --xgb_autotune --level=ms1ms2 --threads=24 try another parameter has the same error：pyprophet score --in=model.osw --level=ms1ms2 --threads=24 root@12c1180257b2:/mnt/data/hela_pd/result2# pyprophet score --in=model.osw --classifier=XGBoost --xgb_autotune --level=ms1ms2 --threads=24 Info: Learn and apply classifier from input data. Warning: Column var_mi_ratio_score contains only invalid/missing values. Column will be dropped. Warning: Column var_elution_model_fit_score contains only invalid/missing values. Column will be dropped. Warning: Column var_sonar_lag contains only invalid/missing values. Column will be dropped. Warning: Column var_sonar_shape contains only invalid/missing values. Column will be dropped. Warning: Column var_sonar_log_sn contains only invalid/missing values. Column will be dropped. Warning: Column var_sonar_log_diff contains only invalid/missing values. Column will be dropped. Warning: Column var_sonar_log_trend contains only invalid/missing values. Column will be dropped. Warning: Column var_sonar_rsq contains only invalid/missing values. Column will be dropped. Info: Data set contains 961149 decoy and 967338 target groups. Info: Summary of input data: Info: 9741919 peak groups Info: 1928487 group ids Info: 35 scores including main score Info: Semi-supervised learning of weights: Info: Start learning on 10 folds using 24 processes. Traceback (most recent call last): File "/usr/local/bin/pyprophet", line 10, in sys.exit(cli()) File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 764, in call return self.main(args, kwargs) File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1163, in invoke rv.append(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 555, in invoke return callback(args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/pyprophet/main.py", line 93, in score PyProphetLearner(infile, outfile, classifier, xgb_hyperparams, xgb_params, xgb_params_space, xeval_fraction, xeval_num_iter, ss_initial_fdr, ss_iteration_fdr, ss_num_iter, ss_main_score, group_id, parametric, pfdr, pi0_lambda, pi0_method, pi0_smooth_df, pi0_smooth_log_pi0, lfdr_truncate, lfdr_monotone, lfdr_transformation, lfdr_adj, lfdr_eps, level, ipf_max_peakgroup_rank, ipf_max_peakgroup_pep, ipf_max_transition_isotope_overlap, ipf_min_transition_sn, tric_chromprob, threads, test).run() File "/usr/local/lib/python3.6/dist-packages/pyprophet/runner.py", line 223, in run (result, scorer, weights) = self.run_algo() File "/usr/local/lib/python3.6/dist-packages/pyprophet/runner.py", line 385, in run_algo (result, scorer, weights) = PyProphet(self.classifier, self.xgb_hyperparams, self.xgb_params, self.xgb_params_space, self.xeval_fraction, self.xeval_num_iter, self.ss_initial_fdr, self.ss_iteration_fdr, self.ss_num_iter, self.group_id, self.parametric, self.pfdr, self.pi0_lambda, self.pi0_method, self.pi0_smooth_df, self.pi0_smooth_log_pi0, self.lfdr_truncate, self.lfdr_monotone, self.lfdr_transformation, self.lfdr_adj, self.lfdr_eps, self.tric_chromprob, self.threads, self.test).learn_and_apply(self.table) File "/usr/local/lib/python3.6/dist-packages/pyprophet/pyprophet.py", line 255, in learn_and_apply result, scorer, trained_weights = self._learn_and_apply(table) File "/usr/local/lib/python3.6/dist-packages/pyprophet/pyprophet.py", line 263, in _learn_and_apply final_classifier = self._learn(experiment) File "/usr/local/lib/python3.6/dist-packages/pyprophet/pyprophet.py", line 295, in _learn res = pool.map(unwrap_self_for_multiprocessing, args) File "/usr/lib/python3.6/multiprocessing/pool.py", line 288, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "/usr/lib/python3.6/multiprocessing/pool.py", line 670, in get raise self._value File "/usr/lib/python3.6/multiprocessing/pool.py", line 450, in _handle_tasks put(task) File "/usr/lib/python3.6/multiprocessing/connection.py", line 206, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/usr/lib/python3.6/multiprocessing/connection.py", line 393, in _send_bytes header = struct.pack("!i", n) struct.error: 'i' format requires -2147483648 <= number <= 2147483647

[uweschmitt]

Looks like your data set is too large!?

[Allen188]

yes, after merge ,model.osw is 8.13GB, Is it because of this? I had run 2GB is successed. Should I change the parameter “subsample_ratio” in step of pyprophet subsample

[grosenberger]

Yes, I would suggest to adjust subsample_ratio to a much smaller value, that should prevent the issue.