Closed mrocklin closed 4 years ago
I took a look at this today and have some feedback/questions. I'm not sure who best can help to answer these, but I suspect that either @randerzander or @mt-jones can help direct to the right people.
Currently we specify the names and dtypes explicitly and use cudf.read_csv. I think that today we might be able to use dask-cudf a bit more naively.
This seems to work
cols = [
"loan_id", "monthly_reporting_period", "servicer", "interest_rate", "current_actual_upb",
"loan_age", "remaining_months_to_legal_maturity", "adj_remaining_months_to_maturity",
"maturity_date", "msa", "current_loan_delinquency_status", "mod_flag", "zero_balance_code",
"zero_balance_effective_date", "last_paid_installment_date", "foreclosed_after",
"disposition_date", "foreclosure_costs", "prop_preservation_and_repair_costs",
"asset_recovery_costs", "misc_holding_expenses", "holding_taxes", "net_sale_proceeds",
"credit_enhancement_proceeds", "repurchase_make_whole_proceeds", "other_foreclosure_proceeds",
"non_interest_bearing_upb", "principal_forgiveness_upb", "repurchase_make_whole_proceeds_flag",
"foreclosure_principal_write_off_amount", "servicing_activity_indicator"
]
performance = dask_cudf.read_csv('/datasets/mortgage/performance/Performance_2000*.txt', delimiter='|', names=cols)
cols = [
'loan_id', 'orig_channel', 'seller_name', 'orig_interest_rate', 'orig_upb', 'orig_loan_term',
'orig_date', 'first_pay_date', 'orig_ltv', 'orig_cltv', 'num_borrowers', 'dti', 'borrower_credit_score',
'first_home_buyer', 'loan_purpose', 'property_type', 'num_units', 'occupancy_status', 'property_state',
'zip', 'mortgage_insurance_percent', 'product_type', 'coborrow_credit_score', 'mortgage_insurance_type',
'relocation_mortgage_indicator'
]
acquisition = dask_cudf.read_csv('/datasets/mortgage/acquisition/Acquisition_*.txt', delimiter='|', names=cols)
Though it may be that there were particular dtypes that we were specifically trying to target. My hope is that the dtype information is no longer necessary (it's sometimes hard to ask users to produce this information in practice)
I actually don't know the context of this computation, but there is something that looks like the following:
def create_ever_features(gdf, **kwargs):
everdf = gdf[['loan_id', 'current_loan_delinquency_status']]
everdf = everdf.groupby('loan_id', method='hash').max()
del(gdf)
everdf['ever_30'] = (everdf['max_current_loan_delinquency_status'] >= 1).astype('int8')
everdf['ever_90'] = (everdf['max_current_loan_delinquency_status'] >= 3).astype('int8')
everdf['ever_180'] = (everdf['max_current_loan_delinquency_status'] >= 6).astype('int8')
everdf.drop_column('max_current_loan_delinquency_status')
return everdf
Which I think we can probably get by with something like the following (on dask-cudf)
everdf = performance[['loan_id', 'current_loan_delinquency_status']]
everdf = everdf.groupby('loan_id').max()
everdf['ever_30'] = (everdf['current_loan_delinquency_status'] >= 1).astype('int8')
everdf['ever_90'] = (everdf['current_loan_delinquency_status'] >= 3).astype('int8')
everdf['ever_180'] = (everdf['current_loan_delinquency_status'] >= 6).astype('int8')
Which raises the following traceback when you call everdf.compute()
Unfortunately I'm not sure what GDFError: GDF_VALIDITY_UNSUPPORTED
means. I'm hoping that @jrhemstad can help explain that error. Unfortunately I don't have a minimal reproducible example here.
Then there is delinquency over time. I'm a bit confused by the use of query here and the repetition. I wonder if there is a way to make this cell a bit smaller while achieving the same result (as well as working on dask-cudf dataframes)
delinq = performance[['loan_id', 'monthly_reporting_period', 'current_loan_delinquency_status']]
delinq_30 = delinq.query('current_loan_delinquency_status >= 1')[['loan_id', 'monthly_reporting_period']].groupby('loan_id').min()
delinq_30['delinquency_30'] = delinq_30['monthly_reporting_period']
# del delinq['monthly_reporting_period']
delinq_90 = delinq.query('current_loan_delinquency_status >= 3')[['loan_id', 'monthly_reporting_period']].groupby('loan_id').min()
delinq_90['delinquency_90'] = delinq_90['monthly_reporting_period']
# del delinq_90['monthly_reporting_period']
delinq_180 = delinq.query('current_loan_delinquency_status >= 6')[['loan_id', 'monthly_reporting_period']].groupby('loan_id').min()
delinq_180['delinquency_180'] = delinq_180['monthly_reporting_period']
# del edelinq_180['monthly_reporting_period']
delinq_merge = delinq_30.merge(delinq_90, how='left', on=['loan_id'])
delinq_merge['delinquency_90'] = delinq_merge['delinquency_90'].fillna(np.dtype('datetime64[ms]').type('1970-01-01').astype('datetime64[ms]'))
delinq_merge = delinq_merge.merge(delinq_180, how='left', on=['loan_id'])
delinq_merge['delinquency_180'] = delinq_merge['delinquency_180'].fillna(np.dtype('datetime64[ms]').type('1970-01-01').astype('datetime64[ms]'))
Here was my attempt
delinq = performance[['loan_id', 'monthly_reporting_period', 'current_loan_delinquency_status']]
d30 = delinq[delinq.current_loan_delinquency_status >= 1].groupby('loan_id').min()
d90 = delinq[delinq.current_loan_delinquency_status >= 3].groupby('loan_id').min()
d180 = delinq[delinq.current_loan_delinquency_status >= 6].groupby('loan_id').min()
I would then probably call dd.concat
to merge these together. However at this point we get an error. It looks like cudf selections don't support bools. I boiled this down and went to raise an issue but found that it had already been raised at https://github.com/rapidsai/cudf/issues/991
A lot of this notebook looks like more of the same. I'm going to pause for now and see if others have some feedback, or if there are others that can continue in this direction.
Also, I should note that I found it much simpler when iterating to drop all of the subprocess and LocalCUDACluster stuff and just work from a single thread on a smaller sample of the data (if you look above you'll see that I'm only looking at the year 2000.
Unfortunately I'm not sure what GDFError: GDF_VALIDITY_UNSUPPORTED means. I'm hoping that @jrhemstad can help explain that error. Unfortunately I don't have a minimal reproducible example here.
This means your "key" columns or your "aggregation" columns have null
values, and the libcudf groupby implementation currently does not support null values. fillna
has been an appropriate workaround in the past.
FYI when folks talk about improving error messages, this is what we mean. The message you just provided would be an excellent (if perhaps sligthly verbose) message. This may or may not be on you though.
If we wanted to not work around this is there an appropriate thing to push on? Can we move this to some issue in rapidsai/cudf?
Ah, sorry, the second statement there was more about null handling, and less about error message handling, which I think we should probably also discuss somewhere other than here if there isn't already a conversation happening.
FYI when folks talk about improving error messages, this is what we mean. The message you just provided would be an excellent (if perhaps sligthly verbose) message. This may or may not be on you though.
At present, all we can do in the C++ side is return an integer-valued error code. This limits the amount of context we can give you without having 10,000 different possible error codes.
That said, this can be improved by throwing exceptions from the C++ side and converting those into Python exceptions. This would allow us to provide more detailed error description strings.
See https://github.com/rapidsai/cudf/issues/862
If we wanted to not work around this is there an appropriate thing to push on? Can we move this to some issue in rapidsai/cudf?
That would be pushing on me to implement this https://github.com/rapidsai/cudf/issues/544
It's on my list as part of a much larger refactor of the groupby implementation. https://github.com/rapidsai/cudf/issues/543
OK great. Thanks for the pointers. Subscribed to those issues (and my apologies for not checking earlier)
Closing as this is stale.
Eventually it would be a nice to use dask-cudf in the E2E notebook. This would be useful both to make the notebook more readable, and also to drive development in cudf/dask-cudf.