search

GeoscienceAustralia / eqrm

Automatically exported from code.google.com/p/eqrm
Other
5 stars 4 forks source link

check scenarios is failing on TS_haz33MR #32

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
What steps will reproduce the problem?
1. Run check scenarios

What do you see instead?
....
*******1002**********
Running scenario TS_haz37.py
*******1002**********
*******1002**********
Running scenario TS_haz36.py
*******1002**********
*******1002**********
Running scenario TS_haz33MR.py
*******1002**********

Traceback (most recent call last):
  File "check_scenarios.py", line 51, in <module>
    check_scenarios_main()
  File "/nas/gemd/georisk_models/earthquake/sandpits/duncan/EQRM/trunk/eqrm_core/eqrm_code/check_scenarios.py", line 539, in check_scenarios_main
    files=files)
  File "/nas/gemd/georisk_models/earthquake/sandpits/duncan/EQRM/trunk/eqrm_core/eqrm_code/check_scenarios.py", line 171, in run_scenarios
    analysis.main(pull_path, True)
  File "/nas/gemd/georisk_models/earthquake/sandpits/duncan/EQRM/trunk/eqrm_core/eqrm_code/analysis.py", line 167, in main
    (event_set, event_activity, source_model) = create_event_set(eqrm_flags, parallel)
  File "/nas/gemd/georisk_models/earthquake/sandpits/duncan/EQRM/trunk/eqrm_core/eqrm_code/event_set.py", line 1561, in create_event_set
    generate_event_set(parallel, eqrm_flags)
  File "/nas/gemd/georisk_models/earthquake/sandpits/duncan/EQRM/trunk/eqrm_core/eqrm_code/event_set.py", line 1503, in generate_event_set
    source_model.save(save_dir)
  File "/nas/gemd/georisk_models/earthquake/sandpits/duncan/EQRM/trunk/eqrm_core/eqrm_code/source_model.py", line 88, in save
    os.mkdir(save_dir)
OSError: [Errno 2] No such file or directory: 
'./implementation_tests/current/TS_haz33MR/current_event_set/source_model'

Original issue reported on code.google.com by duncan.g...@gmail.com on 16 Mar 2012 at 5:39

GoogleCodeExporter commented 9 years ago 
This occurs in Unix and windows.

Original comment by duncan.g...@gmail.com on 16 Mar 2012 at 5:40

GoogleCodeExporter commented 9 years ago 
I ran TS_haz33MR.py individually with;

Then in check scenarios TS_haz31MR.py failed.
I rang TS_haz31MR.py individually and TS_haz30MR.py failed.
So I ran TS_haz30MR.py individually.  
Then check scenarios passed.

Original comment by duncan.g...@gmail.com on 16 Mar 2012 at 6:13

GoogleCodeExporter commented 9 years ago 
I think this is due to the current_event_set dir not being made before trying 
to make the source_model dir.

Original comment by duncan.g...@gmail.com on 16 Mar 2012 at 6:47

GoogleCodeExporter commented 9 years ago 
Event_Set.save and Event_Activity.save both call out to File_Store._save which 
uses the following code to create the save directory and intermediate 
directories if needed

            # Make save dir if necessary
            save_dir = os.path.join(dir, self._name)
            if not os.path.exists(save_dir):
                os.makedirs(save_dir)

Source_Model.save, uses os.mkdir which will raise an exception if the 
intermediate directories are not present.

The interesting thing here is that 
./implementation_tests/current/TS_haz33MR/current_event_set/ should have been 
created by Event_Set.save before Source_Model.save is called, but that does not 
appear to be the case here.

Will make a change to the Source_Model.save method to also create intermediate 
directories if needed.

Original comment by b...@girorosso.com on 18 Mar 2012 at 9:25

GoogleCodeExporter commented 9 years ago

Original comment by b...@girorosso.com on 18 Mar 2012 at 9:26

GoogleCodeExporter commented 9 years ago 
Caused by r999, that is making check_scenarios not use file based arrays by 
default.

The intermediate directories are not created by Event_Set.save and 
Event_Activity.save as there are no file arrays to save:

def _save(self, dir=None):
    """Save the associated .npy files in the given dir."""
    if len(self._array_files) > 0:
        ...

Original comment by b...@girorosso.com on 18 Mar 2012 at 9:29

GoogleCodeExporter commented 9 years ago 
The difference with Source_Model is that the object is pickled to disk rather 
than using numpy's save function.

Original comment by b...@girorosso.com on 18 Mar 2012 at 9:34

GoogleCodeExporter commented 9 years ago 
Revision 1006 contains a temporary fix for this:

- Source_Model save method now creates intermediate directories
- Return event set objects when generating the event set so that the first node
always has the objects, even if they aren't file based.

This means that if SAVE_METHOD is None then running in parallel could cause 
problems. File based arrays are still fine. When SAVE_METHOD is moved into 
eqrm_flags this outstanding issue will be resolved.

Original comment by b...@girorosso.com on 18 Mar 2012 at 10:12