Error when run load_args method from analysis.py after modifying parse_reg_args.py

[tzhang10]

To run the model without specific random seeds, I modify parse_reg_args.py by changing the default value in the line 114~127 from 7 to None. It successfully generated different outputs, however, when I call the load_args() method from analysis.py, this error message showed up: Code changed part:

parser.add_argument("--py_rseed",
                        help="random seed for python, set just before training begins at the top of run_training()",
                        type=int,
                        default=None)

parser.add_argument("--np_rseed",
                        help="random seed for numpy, set just before training begins at the top of run_training()",
                        type=int,
                        default=None)

parser.add_argument("--tf_rseed",
                        help="random seed for tensorflow, set just before training begins at the top of run_training()",
                        type=int,
                        default=None)

input:

log_dirs = "/data/pjiang18/TZ/nn4dms-master/output/training_logs/avgfp/cnns/log_local_local_2024-08-24_01-21-20_avgfp_cnn-2xk13f128_lr0.0001_bs32_UUdC8sb2"
metrics = an.load_args(log_dirs)
metrics

Error message:

usage: ipykernel_launcher.py [-h] [--dataset_name DATASET_NAME]
                             [--dataset_file DATASET_FILE] [--wt_aa WT_AA]
                             [--wt_ofs WT_OFS] [--net_file NET_FILE]
                             [--encoding ENCODING] [--graph_fn GRAPH_FN]
                             [--learning_rate LEARNING_RATE]
                             [--batch_size BATCH_SIZE] [--epochs EPOCHS]
                             [--early_stopping]
                             [--early_stopping_allowance EARLY_STOPPING_ALLOWANCE]
                             [--min_loss_decrease MIN_LOSS_DECREASE]
                             [--split_dir SPLIT_DIR] [--train_size TRAIN_SIZE]
                             [--tune_size TUNE_SIZE] [--test_size TEST_SIZE]
                             [--split_rseed SPLIT_RSEED] [--py_rseed PY_RSEED]
                             [--np_rseed NP_RSEED] [--tf_rseed TF_RSEED]
                             [--step_display_interval STEP_DISPLAY_INTERVAL]
                             [--epoch_checkpoint_interval EPOCH_CHECKPOINT_INTERVAL]
                             [--epoch_evaluation_interval EPOCH_EVALUATION_INTERVAL]
                             [--delete_checkpoints] [--compress_everything]
                             [--log_dir_base LOG_DIR_BASE] [--cluster CLUSTER]
                             [--process PROCESS]
ipykernel_launcher.py: error: argument --py_rseed: invalid int value: 'None'
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
File /packages/apps/jupyter/2023-10-09/lib/python3.11/argparse.py:2529, in ArgumentParser._get_value(self, action, arg_string)
   2528 try:
-> 2529     result = type_func(arg_string)
   2531 # ArgumentTypeErrors indicate errors

ValueError: invalid literal for int() with base 10: 'None'

During handling of the above exception, another exception occurred:

ArgumentError                             Traceback (most recent call last)
File /packages/apps/jupyter/2023-10-09/lib/python3.11/argparse.py:1902, in ArgumentParser.parse_known_args(self, args, namespace)
   1901 try:
-> 1902     namespace, args = self._parse_known_args(args, namespace)
   1903 except ArgumentError as err:

File /packages/apps/jupyter/2023-10-09/lib/python3.11/argparse.py:2114, in ArgumentParser._parse_known_args(self, arg_strings, namespace)
   2113     # consume the next optional and any arguments for it
-> 2114     start_index = consume_optional(start_index)
   2116 # consume any positionals following the last Optional

File /packages/apps/jupyter/2023-10-09/lib/python3.11/argparse.py:2054, in ArgumentParser._parse_known_args.<locals>.consume_optional(start_index)
   2053 for action, args, option_string in action_tuples:
-> 2054     take_action(action, args, option_string)
   2055 return stop

File /packages/apps/jupyter/2023-10-09/lib/python3.11/argparse.py:1962, in ArgumentParser._parse_known_args.<locals>.take_action(action, argument_strings, option_string)
   1961 seen_actions.add(action)
-> 1962 argument_values = self._get_values(action, argument_strings)
   1964 # error if this argument is not allowed with other previously
   1965 # seen arguments, assuming that actions that use the default
   1966 # value don't really count as "present"

File /packages/apps/jupyter/2023-10-09/lib/python3.11/argparse.py:2496, in ArgumentParser._get_values(self, action, arg_strings)
   2495 arg_string, = arg_strings
-> 2496 value = self._get_value(action, arg_string)
   2497 self._check_value(action, value)

File /packages/apps/jupyter/2023-10-09/lib/python3.11/argparse.py:2542, in ArgumentParser._get_value(self, action, arg_string)
   2541     msg = _('invalid %(type)s value: %(value)r')
-> 2542     raise ArgumentError(action, msg % args)
   2544 # return the converted value

ArgumentError: argument --py_rseed: invalid int value: 'None'

During handling of the above exception, another exception occurred:

SystemExit                                Traceback (most recent call last)
    [... skipping hidden 1 frame]

Cell In[5], line 2
      1 log_dirs = "/data/pjiang18/TZ/nn4dms-master/output/training_logs/avgfp/cnns/log_local_local_2024-08-24_01-21-20_avgfp_cnn-2xk13f128_lr0.0001_bs32_UUdC8sb2"
----> 2 metrics = an.load_args(log_dirs)
      3 metrics

File /data/pjiang18/TZ/nn4dms-master/code/analysis.py:98, in load_args(log_dir)
     97 parser = get_parser()
---> 98 args = vars(parser.parse_args(["@{}".format(args_fn)]))  # also convert to a dict
    100 # add the UUID and log_dir to the args

File /packages/apps/jupyter/2023-10-09/lib/python3.11/argparse.py:1869, in ArgumentParser.parse_args(self, args, namespace)
   1868 def parse_args(self, args=None, namespace=None):
-> 1869     args, argv = self.parse_known_args(args, namespace)
   1870     if argv:

File /packages/apps/jupyter/2023-10-09/lib/python3.11/argparse.py:1904, in ArgumentParser.parse_known_args(self, args, namespace)
   1903     except ArgumentError as err:
-> 1904         self.error(str(err))
   1905 else:

File /packages/apps/jupyter/2023-10-09/lib/python3.11/argparse.py:2630, in ArgumentParser.error(self, message)
   2629 args = {'prog': self.prog, 'message': message}
-> 2630 self.exit(2, _('%(prog)s: error: %(message)s\n') % args)

File /packages/apps/jupyter/2023-10-09/lib/python3.11/argparse.py:2617, in ArgumentParser.exit(self, status, message)
   2616     self._print_message(message, _sys.stderr)
-> 2617 _sys.exit(status)

SystemExit: 2

During handling of the above exception, another exception occurred:

AttributeError                            Traceback (most recent call last)
    [... skipping hidden 1 frame]

File /packages/apps/jupyter/2023-10-09/lib/python3.11/site-packages/IPython/core/interactiveshell.py:2097, in InteractiveShell.showtraceback(self, exc_tuple, filename, tb_offset, exception_only, running_compiled_code)
   2094 if exception_only:
   2095     stb = ['An exception has occurred, use %tb to see '
   2096            'the full traceback.\n']
-> 2097     stb.extend(self.InteractiveTB.get_exception_only(etype,
   2098                                                      value))
   2099 else:
   2101     def contains_exceptiongroup(val):

File /packages/apps/jupyter/2023-10-09/lib/python3.11/site-packages/IPython/core/ultratb.py:710, in ListTB.get_exception_only(self, etype, value)
    702 def get_exception_only(self, etype, value):
    703     """Only print the exception type and message, without a traceback.
    704 
    705     Parameters
   (...)
    708     value : exception value
    709     """
--> 710     return ListTB.structured_traceback(self, etype, value)

File /packages/apps/jupyter/2023-10-09/lib/python3.11/site-packages/IPython/core/ultratb.py:568, in ListTB.structured_traceback(self, etype, evalue, etb, tb_offset, context)
    565     chained_exc_ids.add(id(exception[1]))
    566     chained_exceptions_tb_offset = 0
    567     out_list = (
--> 568         self.structured_traceback(
    569             etype,
    570             evalue,
    571             (etb, chained_exc_ids),  # type: ignore
    572             chained_exceptions_tb_offset,
    573             context,
    574         )
    575         + chained_exception_message
    576         + out_list)
    578 return out_list

File /packages/apps/jupyter/2023-10-09/lib/python3.11/site-packages/IPython/core/ultratb.py:1435, in AutoFormattedTB.structured_traceback(self, etype, evalue, etb, tb_offset, number_of_lines_of_context)
   1433 else:
   1434     self.tb = etb
-> 1435 return FormattedTB.structured_traceback(
   1436     self, etype, evalue, etb, tb_offset, number_of_lines_of_context
   1437 )

File /packages/apps/jupyter/2023-10-09/lib/python3.11/site-packages/IPython/core/ultratb.py:1326, in FormattedTB.structured_traceback(self, etype, value, tb, tb_offset, number_of_lines_of_context)
   1323 mode = self.mode
   1324 if mode in self.verbose_modes:
   1325     # Verbose modes need a full traceback
-> 1326     return VerboseTB.structured_traceback(
   1327         self, etype, value, tb, tb_offset, number_of_lines_of_context
   1328     )
   1329 elif mode == 'Minimal':
   1330     return ListTB.get_exception_only(self, etype, value)

File /packages/apps/jupyter/2023-10-09/lib/python3.11/site-packages/IPython/core/ultratb.py:1173, in VerboseTB.structured_traceback(self, etype, evalue, etb, tb_offset, number_of_lines_of_context)
   1164 def structured_traceback(
   1165     self,
   1166     etype: type,
   (...)
   1170     number_of_lines_of_context: int = 5,
   1171 ):
   1172     """Return a nice text document describing the traceback."""
-> 1173     formatted_exception = self.format_exception_as_a_whole(etype, evalue, etb, number_of_lines_of_context,
   1174                                                            tb_offset)
   1176     colors = self.Colors  # just a shorthand + quicker name lookup
   1177     colorsnormal = colors.Normal  # used a lot

File /packages/apps/jupyter/2023-10-09/lib/python3.11/site-packages/IPython/core/ultratb.py:1063, in VerboseTB.format_exception_as_a_whole(self, etype, evalue, etb, number_of_lines_of_context, tb_offset)
   1060 assert isinstance(tb_offset, int)
   1061 head = self.prepare_header(str(etype), self.long_header)
   1062 records = (
-> 1063     self.get_records(etb, number_of_lines_of_context, tb_offset) if etb else []
   1064 )
   1066 frames = []
   1067 skipped = 0

File /packages/apps/jupyter/2023-10-09/lib/python3.11/site-packages/IPython/core/ultratb.py:1131, in VerboseTB.get_records(self, etb, number_of_lines_of_context, tb_offset)
   1129 while cf is not None:
   1130     try:
-> 1131         mod = inspect.getmodule(cf.tb_frame)
   1132         if mod is not None:
   1133             mod_name = mod.__name__

AttributeError: 'tuple' object has no attribute 'tb_frame'

Could you please give me some insights on how to solve this issue?

[samgelman]

This error arises because an.load_args(log_dirs) tries to read the training arguments, which were saved in a text file during training, and re-parse using argparse. The problem is the random seed arguments are specified as type int, but the value None is read in from the text file as a string.

I believe this fix should work, but I have not tested it.

First, define a new function int_or_none:

def int_or_none(value):
    if value == "None":
        return None
    return int(value)

Then, specify this function as the type for the argparse arguments:

parser.add_argument("--py_rseed",
                        help="random seed for python, set just before training begins at the top of run_training()",
                        type=int_or_none,
                        default=None)

Let me know if that works!

[tzhang10]

Thanks so much for the help! It works!