search

nredell / ShapML.jl

A Julia package for interpretable machine learning with stochastic Shapley values
https://nredell.github.io/ShapML.jl/dev/
MIT License
85 stars 7 forks source link

Support for DataFrames 22.0? #7

Open rokapre opened 3 years ago

rokapre commented 3 years ago

Seems like this is not compatible with the latest DataFrames 22.0 version if you do add ShapML. I was able to download it from the github repository but it resulted in downgrading to DataFrames 21.8. There are quite a few changes from 21.8 to 22.0.

nredell commented 3 years ago

I just merged a pull request that addresses some (all?) of this. Doing some minor cleanup, testing, and will push a new version soon.

rokapre commented 3 years ago

Has it been pushed yet? I recently tried to add the package but the usual ] add ShapML didn't work as I got a bunch of dependency errors with CategoricalArrays, DataFrames etc. Then I added via the github and DataFrames still got downgraded

casasgomezuribarri commented 3 years ago

I have a similar problem when trying to add the package with ] add ShapML (ERROR: Unsatisfiable requirements detected for package CategoricalArrays [324d7699]:..., see below). It is a long error message mentioning several packages from the MLJ family (MLJ, MLJBase, MLJTuning, MLJMultivariateStatsInterface... etc), as well as others like Distributions, Tables and DataFrames. However, DataFrames is the only one in conflict directly with ShapML (see near the end of the error message).

I assume my problem is related, if not the same one as the poster of this issue.

ERROR: LoadError: Unsatisfiable requirements detected for package CategoricalArrays [324d7699]: 
> ERROR: LoadError: Unsatisfiable requirements detected for package CategoricalArrays [324d7699]:
>  CategoricalArrays [324d7699] log:
>  ├─possible versions are: [0.3.11, 0.3.13-0.3.14, 0.4.0, 0.5.0-0.5.5, 0.6.0, 0.7.0-0.7.7, 0.8.0-0.8.3, 0.9.0] or uninstalled
>  ├─restricted by compatibility requirements with MLJ [add582a8] to versions: [0.3.11, 0.3.13-0.3.14, 0.4.0, 0.5.0-0.5.5, 0.6.0, 0.7.0-0.7.7, 0.8.0-0.8.3, 0.9.0]
>  │ └─MLJ [add582a8] log:
>  │   ├─possible versions are: [0.1.0-0.1.1, 0.2.0-0.2.5, 0.3.0, 0.4.0, 0.5.0-0.5.9, 0.6.0-0.6.1, 0.7.0, 0.8.0, 0.9.0-0.9.3, 0.10.0-0.10.3, 0.11.0-0.11.6, 0.12.0-0.12.1, 0.13.0, 0.14.0-0.14.1, 0.15.0-0.15.2] or uninstalled   
>  │   ├─restricted to versions * by an explicit requirement, leaving only versions [0.1.0-0.1.1, 0.2.0-0.2.5, 0.3.0, 0.4.0, 0.5.0-0.5.9, 0.6.0-0.6.1, 0.7.0, 0.8.0, 0.9.0-0.9.3, 0.10.0-0.10.3, 0.11.0-0.11.6, 0.12.0-0.12.1, 0.13.0, 0.14.0-0.14.1, 0.15.0-0.15.2]
>  │   ├─restricted by compatibility requirements with MLJBase [a7f614a8] to versions: [0.7.0, 0.8.0, 0.9.0-0.9.3, 0.10.0-0.10.3, 0.11.0-0.11.6, 0.12.0-0.12.1, 0.13.0, 0.14.0-0.14.1, 0.15.0-0.15.2] or uninstalled, leaving only versions: [0.7.0, 0.8.0, 0.9.0-0.9.3, 0.10.0-0.10.3, 0.11.0-0.11.6, 0.12.0-0.12.1, 0.13.0, 0.14.0-0.14.1, 0.15.0-0.15.2]
>  │   │ └─MLJBase [a7f614a8] log:
>  │   │   ├─possible versions are: [0.1.0-0.1.1, 0.2.0-0.2.6, 0.3.0, 0.4.0, 0.5.0, 0.6.0, 0.7.0-0.7.5, 0.8.0-0.8.4, 0.9.0-0.9.2, 0.10.0-0.10.1, 0.11.0-0.11.10, 0.12.0-0.12.8, 0.13.0-0.13.10, 0.14.0-0.14.9, 0.15.0-0.15.7, 0.16.0-0.16.3, 0.17.0-0.17.1] or uninstalled
>  │   │   ├─restricted by compatibility requirements with MLJTuning [03970b2e] to versions: [0.10.1, 0.11.0-0.11.10, 0.12.0-0.12.8, 0.13.0-0.13.10, 0.14.0-0.14.9, 0.15.0-0.15.7, 0.16.0-0.16.3, 0.17.0-0.17.1]
>  │   │   │ └─MLJTuning [03970b2e] log:
>  │   │   │   ├─possible versions are: [0.1.0-0.1.4, 0.2.0, 0.3.0-0.3.6, 0.4.0-0.4.3, 0.5.0-0.5.5, 0.6.0-0.6.1] or uninstalled
>  │   │   │   ├─restricted to versions * by an explicit requirement, leaving only versions [0.1.0-0.1.4, 0.2.0, 0.3.0-0.3.6, 0.4.0-0.4.3, 0.5.0-0.5.5, 0.6.0-0.6.1]
>  │   │   │   └─restricted by compatibility requirements with MLJBase [a7f614a8] to versions: [0.1.0-0.1.4, 0.2.0, 0.3.0-0.3.6, 0.4.0-0.4.3, 0.5.0-0.5.5, 0.6.0] or uninstalled, leaving only versions: [0.1.0-0.1.4, 0.2.0, 0.3.0-0.3.6, 0.4.0-0.4.3, 0.5.0-0.5.5, 0.6.0]
>  │   │   │     └─MLJBase [a7f614a8] log: see above
>  │   │   ├─restricted by compatibility requirements with MLJ [add582a8] to versions: [0.1.0-0.1.1, 0.2.0-0.2.6, 
> 0.4.0, 0.5.0, 0.6.0, 0.7.0-0.7.5, 0.8.1-0.8.4, 0.9.1-0.9.2, 0.10.0-0.10.1, 0.11.0-0.11.10, 0.12.0-0.12.8, 0.13.1-0.13.10, 0.14.0-0.14.9, 0.15.1-0.15.7, 0.16.0-0.16.3], leaving only versions: [0.10.1, 0.11.0-0.11.10, 0.12.0-0.12.8, 0.13.1-0.13.10, 0.14.0-0.14.9, 0.15.1-0.15.7, 0.16.0-0.16.3]
>  │   │   │ └─MLJ [add582a8] log: see above
>  │   │   └─restricted by compatibility requirements with Distributions [31c24e10] to versions: [0.12.4-0.12.8, 0.13.0-0.13.10, 0.14.0-0.14.9, 0.15.0-0.15.7, 0.16.0-0.16.3, 0.17.0-0.17.1] or uninstalled, leaving only versions: [0.12.4-0.12.8, 0.13.1-0.13.10, 0.14.0-0.14.9, 0.15.1-0.15.7, 0.16.0-0.16.3]
>  │   │     └─Distributions [31c24e10] log:
>  │   │       ├─possible versions are: [0.16.0-0.16.4, 0.17.0, 0.18.0, 0.19.1-0.19.2, 0.20.0, 0.21.0-0.21.3, 0.21.5-0.21.12, 0.22.0-0.22.6, 0.23.0-0.23.12, 0.24.0-0.24.12] or uninstalled
>  │   │       └─restricted by compatibility requirements with MLJGLMInterface [caf8df21] to versions: [0.23.0-0.23.12, 0.24.0-0.24.12]
>  │   │         └─MLJGLMInterface [caf8df21] log:
>  │   │           ├─possible versions are: 0.1.0-0.1.3 or uninstalled
>  │   │           └─restricted to versions * by an explicit requirement, leaving only versions 0.1.0-0.1.3       
>  │   ├─restricted by compatibility requirements with Distributions [31c24e10] to versions: [0.10.2-0.10.3, 0.11.0-0.11.6, 0.12.0-0.12.1, 0.13.0, 0.14.0-0.14.1, 0.15.0-0.15.2] or uninstalled, leaving only versions: [0.10.2-0.10.3, 0.11.0-0.11.6, 0.12.0-0.12.1, 0.13.0, 0.14.0-0.14.1, 0.15.0-0.15.2]
>  │   │ └─Distributions [31c24e10] log: see above
>  │   └─restricted by compatibility requirements with MLJModels [d491faf4] to versions: [0.12.0-0.12.1, 0.13.0, 0.14.0-0.14.1, 0.15.0-0.15.2] or uninstalled, leaving only versions: [0.12.0-0.12.1, 0.13.0, 0.14.0-0.14.1, 0.15.0-0.15.2]
>  │     └─MLJModels [d491faf4] log:
>  │       ├─possible versions are: [0.1.0-0.1.1, 0.2.0-0.2.5, 0.3.0, 0.4.0, 0.5.0-0.5.9, 0.6.0-0.6.3, 0.7.0-0.7.2, 0.8.0-0.8.4, 0.9.0-0.9.12, 0.10.0-0.10.1, 0.11.0-0.11.3, 0.12.0-0.12.11, 0.13.0-0.13.3] or uninstalled        
>  │       ├─restricted by compatibility requirements with MLJ [add582a8] to versions: [0.1.0-0.1.1, 0.2.0-0.2.5, 
> 0.3.0, 0.4.0, 0.5.0-0.5.9, 0.6.0-0.6.3, 0.7.0-0.7.2, 0.8.0-0.8.4, 0.9.0-0.9.12, 0.10.0-0.10.1, 0.11.0-0.11.3, 0.12.1-0.12.11, 0.13.0-0.13.3]
>  │       │ └─MLJ [add582a8] log: see above
>  │       ├─restricted by compatibility requirements with MLJBase [a7f614a8] to versions: [0.7.0-0.7.2, 0.8.0-0.8.4, 0.9.0-0.9.12, 0.10.0-0.10.1, 0.11.0-0.11.3, 0.12.0-0.12.11, 0.13.0-0.13.3] or uninstalled, leaving only versions: [0.7.0-0.7.2, 0.8.0-0.8.4, 0.9.0-0.9.12, 0.10.0-0.10.1, 0.11.0-0.11.3, 0.12.1-0.12.11, 0.13.0-0.13.3]     
>  │       │ └─MLJBase [a7f614a8] log: see above
>  │       ├─restricted by compatibility requirements with Distributions [31c24e10] to versions: [0.9.1-0.9.12, 0.10.0-0.10.1, 0.11.0-0.11.3, 0.12.0-0.12.11, 0.13.0-0.13.3] or uninstalled, leaving only versions: [0.9.1-0.9.12, 0.10.0-0.10.1, 0.11.0-0.11.3, 0.12.1-0.12.11, 0.13.0-0.13.3]
>  │       │ └─Distributions [31c24e10] log: see above
>  │       └─restricted by compatibility requirements with Distances [b4f34e82] to versions: [0.11.0-0.11.3, 0.12.0-0.12.11, 0.13.0-0.13.3] or uninstalled, leaving only versions: [0.11.0-0.11.3, 0.12.1-0.12.11, 0.13.0-0.13.3] 
>  │         └─Distances [b4f34e82] log:
>  │           ├─possible versions are: [0.7.0-0.7.4, 0.8.0-0.8.2, 0.9.0-0.9.2, 0.10.0-0.10.2] or uninstalled     
>  │           └─restricted by compatibility requirements with MLJMultivariateStatsInterface [1b6a4a23] to versions: [0.9.0-0.9.2, 0.10.0-0.10.2]
>  │             └─MLJMultivariateStatsInterface [1b6a4a23] log:
>  │               ├─possible versions are: 0.1.0-0.1.7 or uninstalled
>  │               └─restricted to versions * by an explicit requirement, leaving only versions 0.1.0-0.1.7       
>  ├─restricted by compatibility requirements with MLJBase [a7f614a8] to versions: [0.7.3-0.7.7, 0.8.0-0.8.3, 0.9.0]
>  │ └─MLJBase [a7f614a8] log: see above
>  ├─restricted by compatibility requirements with DataFrames [a93c6f00] to versions: 0.7.0-0.7.7, leaving only versions: 0.7.3-0.7.7
>  │ └─DataFrames [a93c6f00] log:
>  │   ├─possible versions are: [0.11.7, 0.12.0, 0.13.0-0.13.1, 0.14.0-0.14.1, 0.15.0-0.15.2, 0.16.0, 0.17.0-0.17.1, 0.18.0-0.18.4, 0.19.0-0.19.4, 0.20.0-0.20.2, 0.21.0-0.21.8, 0.22.0-0.22.5] or uninstalled
>  │   ├─restricted by compatibility requirements with ShapML [8523bd24] to versions: 0.20.0-0.20.2
>  │   │ └─ShapML [8523bd24] log:
>  │   │   ├─possible versions are: [0.1.0, 0.2.0, 0.3.0] or uninstalled
>  │   │   └─restricted to versions * by an explicit requirement, leaving only versions [0.1.0, 0.2.0, 0.3.0]     
>  │   └─restricted by compatibility requirements with Tables [bd369af6] to versions: [0.11.7, 0.12.0, 0.13.0-0.13.1, 0.20.1-0.20.2, 0.21.0-0.21.8, 0.22.0-0.22.5] or uninstalled, leaving only versions: 0.20.1-0.20.2
>  │     └─Tables [bd369af6] log:
>  │       ├─possible versions are: [0.1.0-0.1.15, 0.1.17-0.1.19, 0.2.0-0.2.11, 1.0.0-1.0.5, 1.1.0, 1.2.0-1.2.2, 1.3.0-1.3.1] or uninstalled
>  │       └─restricted by compatibility requirements with MLJGLMInterface [caf8df21] to versions: [1.1.0, 1.2.0-1.2.2, 1.3.0-1.3.1]
>  │         └─MLJGLMInterface [caf8df21] log: see above
>  └─restricted by compatibility requirements with MLJModels [d491faf4] to versions: [0.8.0-0.8.3, 0.9.0] — no versions left
>    └─MLJModels [d491faf4] log: see above
4SAnalyticsnModelling commented 3 years ago

Try adding ShapML as ]add "https://github.com/nredell/ShapML.jl". Add all the other packages before adding ShapML.jl so that the package can downgrade the required packages as needed. That's a work around working for me so far. I have also been able to work with it by installing ShapML in a separate environment, and all other ML packages in the base environment. In all cases I added the ShapML.jl with the git repo URL as mentioned above.

casasgomezuribarri commented 3 years ago

Thanks for your help. I got it working on the same environment with]add ShapML#master, which I assume triggers the same behaviour?

4SAnalyticsnModelling commented 3 years ago

Yes, it does, unfortunately. It will still downgrade some of the packages including DataFrames. But that's the only work around I have for now until nredell pushes the updated version of ShapML. Thank you!

nredell commented 3 years ago

I'm a bit hit or miss on GitHub nowadays--maybe slowly getting back into it, though--, but now seems like a good time to actually fix this. The newest version of DataFrames has broken a lot more than the join functions.

Creating a data frame from a matrix is deprecated--which occurs in several spots--, and column names are getting lost somewhere along the way. A unit test has been failing for exactly this reason since the last pull request.

In progress.

4SAnalyticsnModelling commented 3 years ago

Thanks for looking into it! Yes you are right, things are breaking at several places. I have tried using DataFrames.combine and DataFrames.groupby functions in predict.jl instead of the custom _aggregate function. That passed the test with the most updated version of DataFrames.

4SAnalyticsnModelling commented 3 years ago

@nredell I have created 2 pull requests on proposed changes in predict.jl and aggregate.jl. Those changes gave me clean run without any deprecation warning. All tests were passed with the proposed changes. Please have a look at those proposed changes.

4SAnalyticsnModelling commented 3 years ago

@nredell with those proposed changes you can bump up the DataFrames.jl compatibility to its most updated version.

pdeffebach commented 3 years ago

Bumping this, as now we are at DataFrames 1.0. Let me know if you need help with a PR to get support 1.0.