Python example in exercise 1 errors

[kevinykuo]

Stack trace below. Note I'm running on mac but this might not be OS specific.

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
~/Dropbox/Projects/Workshop2019/oasis1/lib/python3.6/site-packages/oasislmf/model_preparation/il_inputs.py in get_il_input_items(***failed resolving arguments***)
    330             ([SOURCE_IDX['acc']] if SOURCE_IDX['acc'] in il_inputs_df else []) +
--> 331             site_pd_and_site_all_term_cols +
    332             term_cols

AttributeError: 'Index' object has no attribute 'to_list'

The above exception was the direct cause of the following exception:

OasisException                            Traceback (most recent call last)
<ipython-input-9-7f686709f916> in <module>
     16     (gul_losses, il_losses, ri_losses) = om.run_deterministic(
     17         src_dir='/tmp/exercise_1_oed',
---> 18         loss_percentage_of_tiv=i/100)
     19     total_loss = il_losses.loss.sum()
     20 

~/Dropbox/Projects/Workshop2019/oasis1/lib/python3.6/site-packages/oasislmf/utils/log.py in wrapper(*args, **kwargs)
    104 
    105             start = time.time()
--> 106             result = func(*args, **kwargs)
    107             end = time.time()
    108             logger.info(

~/Dropbox/Projects/Workshop2019/oasis1/lib/python3.6/site-packages/oasislmf/manager.py in run_deterministic(self, src_dir, run_dir, loss_percentage_of_tiv, alloc_rule, net_ri)
    760             accounts_fp=accounts_fp,
    761             ri_info_fp=ri_info_fp,
--> 762             ri_scope_fp=ri_scope_fp
    763         )
    764 

~/Dropbox/Projects/Workshop2019/oasis1/lib/python3.6/site-packages/oasislmf/utils/log.py in wrapper(*args, **kwargs)
    104 
    105             start = time.time()
--> 106             result = func(*args, **kwargs)
    107             end = time.time()
    108             logger.info(

~/Dropbox/Projects/Workshop2019/oasis1/lib/python3.6/site-packages/oasislmf/manager.py in generate_oasis_files(self, target_dir, exposure_fp, exposure_profile, exposure_profile_fp, keys_fp, lookup_config, lookup_config_fp, keys_data_fp, model_version_fp, lookup_package_fp, complex_lookup_config_fp, user_data_dir, supported_oed_coverage_types, summarise_exposure, accounts_fp, accounts_profile, accounts_profile_fp, fm_aggregation_profile, fm_aggregation_profile_fp, ri_info_fp, ri_scope_fp, oasis_files_prefixes)
    458             exposure_profile=exposure_profile,
    459             accounts_profile=accounts_profile,
--> 460             fm_aggregation_profile=fm_aggregation_profile
    461         )
    462 

~/Dropbox/Projects/Workshop2019/oasis1/lib/python3.6/site-packages/oasislmf/utils/log.py in wrapper(*args, **kwargs)
    104 
    105             start = time.time()
--> 106             result = func(*args, **kwargs)
    107             end = time.time()
    108             logger.info(

~/Dropbox/Projects/Workshop2019/oasis1/lib/python3.6/site-packages/oasislmf/model_preparation/il_inputs.py in get_il_input_items(***failed resolving arguments***)
    532 
    533     except (AttributeError, KeyError, IndexError, TypeError, ValueError) as e:
--> 534         raise OasisException from e
    535 
    536     return il_inputs_df, accounts_df

OasisException: 

[mpinkerton-oasis]

Which cell were you running?

[kevinykuo]

# Run the FM directly via Python code.

# This can enable more complex analytical operations. In the example below, we 
# estimate the damage level (as a % of TIV) at which the portfolio attachs and exhausts.

from oasislmf.manager import OasisManager
om = OasisManager()

# At what level of loss do the accounts attach?
last_loss = 0
attachment = 0
exhaustion = 0
# Cycle through all percentage loss levels in 1% increments
for i in range (0, 100):
    # Generate the losses
    (gul_losses, il_losses, ri_losses) = om.run_deterministic(
        src_dir='/tmp/exercise_1_oed', 
        loss_percentage_of_tiv=i/100)
    total_loss = il_losses.loss.sum()

    # Has the portfolio attached?
    if attachment == 0 and total_loss > 0:
        attachment = i

    # Has the portfolio exhasuted?
    if i > 0 and total_loss > 0 and total_loss == last_loss:
        exhaustion = i
        break

    last_loss = total_loss

print("Attaching loss level: {}%; Exhausting loss level: {}%".format(attachment, exhaustion))

[mpinkerton-oasis]

@sr-murthy have you encountered any similar issues with the MDK on Mac?

[sr-murthy]

I have never seen this error because I haven't run any of these notebooks. Which notebook/exercise does this relate to, and which step?

[mpinkerton-oasis]

I meant more generally, but it should be easy to run. As per note above, it is exercise 1 and I think the last cell.

[sr-murthy]

No, this error hasn't come up in any other context of running the MDK.

[sr-murthy]

The last step in exercise 1 could well cause a problem because of the starting value of the for loop

for i in range (0, 100):
# Generate the losses
(gul_losses, il_losses, ri_losses) = om.run_deterministic(
    src_dir='/tmp/exercise_1_oed', 
    loss_percentage_of_tiv=i/100)
total_loss = il_losses.loss.sum()

If i is 0 then loss_percentage_of_tiv is 0 in the manager's generate_deterministic_losses method, which means that all the initially created GUL items will have loss values of 0

https://github.com/OasisLMF/OasisLMF/blob/develop/oasislmf/manager.py#L641

As only GUL items with non-zero losses are selected, with a loss factor of 0 there wouldn't be any final stage GUL items, which means empty dataframes and indexing errors.

Perhaps this is the problem, but I don't know for sure. I would have to reproduce the problem.

[sr-murthy]

@kevinykuo Please try exercise 1 again. I've made a small change to the for loop in the last step.

[mpinkerton-oasis]

Did you replicate this issue? I haven't had any problem running with loss-factor=0, and this should probably be allowed and just create zero losses.

[sr-murthy]

I haven't been able to replicate it because of issues with Jupyter - it seems unable to find oasislmf in the notebook environment. I tried to add the usual location of site-packages in the virtual env. to sys.path in the notebook directly, but that didn't work.

[kevinykuo]

Still hitting the same error with the update.

[sr-murthy]

I will have another look at it.

[sr-murthy]

@kevinykuo Please try again - the problem appears to have been related to a typo in the oasislmf package Git branch URI in the original version of the requirements file. I have fixed this, and it should now install the develop branch rather than the release.

I have managed to complete all the exercise 1 steps, including the last one (I added some extra print statements in the for loop for debugging, but this can be removed). Here's the output of the last step

Generating losses using loss factor 0.0: no attachment found
Generating losses using loss factor 0.01: no attachment found
Generating losses using loss factor 0.02: no attachment found
Generating losses using loss factor 0.03: no attachment found
Generating losses using loss factor 0.04: no attachment found
Generating losses using loss factor 0.05: no attachment found
Generating losses using loss factor 0.06: no attachment found
Generating losses using loss factor 0.07: no attachment found
Generating losses using loss factor 0.08: no attachment found
Generating losses using loss factor 0.09: found attachment
....
Attaching loss level: 9%; Exhausting loss level: 27%