search

LinkageIO / Camoco

Camoco is a fully-fledged software package for building co-expression networks and analyzing the overlap interactions among genes.
MIT License
41 stars 32 forks source link

odo now breaks with the newest pandas due to tslib deprecation #88

Closed monprin closed 5 years ago

monprin commented 5 years ago

I'm getting the following error when trying to run with pandas>=0.24:

Traceback (most recent call last):
  File "MaizeBuildCommands.py", line 5, in <module>
    import camoco as co
  File "/usr/local/lib/python3.7/site-packages/camoco/__init__.py", line 53, in <module>
    from .COB import COB
  File "/usr/local/lib/python3.7/site-packages/camoco/COB.py", line 38, in <module>
    from odo import odo
  File "/usr/local/lib/python3.7/site-packages/odo/__init__.py", line 29, in <module>
    from .backends.pandas import pd
  File "/usr/local/lib/python3.7/site-packages/odo/backends/pandas.py", line 94, in <module>
    @convert.register((pd.Timestamp, pd.Timedelta), (pd.tslib.NaTType, type(None)))
AttributeError: module 'pandas' has no attribute 'tslib'

The time has officially come to eliminate the dask stuff, at the very least odo, since it only seems to appear once in COB.

I can look into at least doing a PR for the odo problem next week and getting a better handle on what other dask stuff is left and how it can be eliminated.

Solution in the meantime is stick to pandas 0.23.x

schae234 commented 5 years ago

Installing hdf5 is now possible via pip ... We could migrate back to that ... Is there any big benefit in using bcolz over hdf5?

On Fri, Feb 1, 2019, 11:56 AM Joe Jeffers notifications@github.com wrote:

I'm getting the following error when trying to run with pandas>=0.24:

Traceback (most recent call last): File "MaizeBuildCommands.py", line 5, in import camoco as co File "/usr/local/lib/python3.7/site-packages/camoco/init.py", line 53, in from .COB import COB File "/usr/local/lib/python3.7/site-packages/camoco/COB.py", line 38, in from odo import odo File "/usr/local/lib/python3.7/site-packages/odo/init.py", line 29, in from .backends.pandas import pd File "/usr/local/lib/python3.7/site-packages/odo/backends/pandas.py", line 94, in @convert.register((pd.Timestamp, pd.Timedelta), (pd.tslib.NaTType, type(None))) AttributeError: module 'pandas' has no attribute 'tslib'

The time has officially come to eliminate the dask stuff, at the very least odo, since it only seems to appear once in COB.

I can look into at least doing a PR for the odo problem next week and getting a better handle on what other dask stuff is left and how it can be eliminated.

Solution in the meantime is stick to pandas 0.23.x

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/LinkageIO/Camoco/issues/88, or mute the thread https://github.com/notifications/unsubscribe-auth/ACcpvy2vjMaoCM4yCG3xRmQoK97iwUJzks5vJH_KgaJpZM4aey8E .

--

Rob Schaefer, PhD Linkage Analytics, LLC

http://linkage.io http://github.com/LinkageIO

monprin commented 5 years ago

Hmm, I'm trying to remember back to the issues we talked about when moving away from it in the first place. I think the main things that we were worried about was that it was closed source and the painful installation process. And then when we moved from feather, we worried about the chunking. So if you can now install it with pip then I think it would be fine.

The main problem I envision would be the necessity of recomputing existing datasets, but if there was a built in converter that was triggered when older bcolz files were detected to just install bcolz, do the conversion and clean it all up, that could make it pretty painless. And then in a couple versions it could all be fully deprecated with a notice to install an older version to do the conversion.

I don't think I will necessarily have time to take that on, but I am up and running on cob again and have the camoco docker image working well. So I will hopefully be finishing that off in the next couple weeks (I know this is like the third deadline I have blown for but moving cross-country was more time consuming than I thought, who knew). Do you want to take this issue on then and just add the pandas < 0.24 req for now and I will just leave it alone?

schae234 commented 5 years ago

Seems like a "2.0" issue if we want to break backwards compatibility. Let's focus on bringing cob up to speed with Camoco and just fix the dependency to specify pandas <0.24

On Fri, Feb 1, 2019, 12:32 PM Joe Jeffers notifications@github.com wrote:

Hmm, I'm trying to remember back to the issues we talked about when moving away from it in the first place. I think the main things that we were worried about was that it was closed source and the painful installation process. And then when we moved from feather, we worried about the chunking. So if you can now install it with pip then I think it would be fine.

The main problem I envision would be the necessity of recomputing existing datasets, but if there was a built in converter that was triggered when older bcolz files were detected to just install bcolz, do the conversion and clean it all up, that could make it pretty painless. And then in a couple versions it could all be fully deprecated with a notice to install an older version to do the conversion.

I don't think I will necessarily have time to take that on, but I am up and running on cob again and have the camoco docker image working well. So I will hopefully be finishing that off in the next couple weeks (I know this is like the third deadline I have blown for but moving cross-country was more time consuming than I thought, who knew). Do you want to take this issue on then and just add the pandas < 0.24 req for now and I will just leave it alone?

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/LinkageIO/Camoco/issues/88#issuecomment-459822074, or mute the thread https://github.com/notifications/unsubscribe-auth/ACcpv2n96sj7966xR_ONv6v1cHahRbH9ks5vJIg5gaJpZM4aey8E .

--

Rob Schaefer, PhD Linkage Analytics, LLC

http://linkage.io http://github.com/LinkageIO

monprin commented 5 years ago

Yes COB is close, I have it booting up with 0.6.1, there are just a couple little bugs and optimizations for stability I have left to do.

As for bcolz, if it does not change compatibility at all, would you accept a PR to eliminate the blaze+odo dependency for the next point release?

After a quick look there is very little that actually depends on those two anymore and what does could be pretty easily replaced with the raw bcolz API it seems. It could be a stop gap that maintained new pandas compat until you had time to do the full numeric storage overhaul.

schae234 commented 5 years ago

looks like they are deprecating indexing using pd.DataFrame.ix too, which is all over our codebase. Shouldn't be a hard fix though, but might as well do it all at the same time.

monprin commented 5 years ago

Okay sounds good, just for reference, when I was looking to see how hard it would be fix it, I made these notes on how to replace the relevant calls in COB.py to remove the dependence:

_bcolz(blaze=True) => _bcolz(raw=True)
158: ctable shape
213: carray wheretrue
245-259: Already being run directly on bcolz object, remove abstraction
293,301: Already being run directly on bcolz object, remove abstraction
1635: ctable fetchwhere <numexpr expression testing for nan> out_flavor='numpy', out_cols='score'
1839: ctable fetchwhere <numexpr expression testing for sigs> out_flavor='numpy'
1867: ctable shape
1872: ctable todataframe to ndarray
1807: Fix the _raw_coex mess

I'm sure they are not all quite right, but should be a pretty good start. Also 1807 refers to an extra function in the main Camoco.py, That should really just be removed and done in place in COB.py since it is only used once anyway.

If you have any question or want me to look at it more, let me know. FYI I'm setting aside time this weekend for COB, so with a little luck I should get pretty close to done.

TCx64 commented 5 years ago

Hello,

I'm trying to install Camoco for a Researcher here at Colorado State University. I have the same output that Joe posted at the top of this issue thread.. Is there a way to force the pip install camoco to use pandas-0.23.X as you mention above, instead of pandas-0.24.1?

From the middle of the installation process:

Collecting pandas>=0.19.2 (from camoco) Downloading https://files.pythonhosted.org/packages/e6/de/a0d3defd8f338eaf53ef716e40ef6d6c277c35d50e09b586e170169cdf0d/**pandas-0.24.1**-cp36-cp36m-manylinux1_x86_64.whl (10.1MB)

Thanks! Tyson

schae234 commented 5 years ago

Hi Tyson,

I thought I had this fixed, but I forgot one step. I can have the fix on pip in like a day, but if you follow the second set of instructions to install the development version it should go.

Sorry about that, please let me know if anything else pops up or how things turn out!

Rob

On Fri, Feb 8, 2019, 3:38 PM TCx64 notifications@github.com wrote:

Hello,

I'm trying to install Camoco for a Researcher here at Colorado State University. I have the same output that Joe posted at the top of this issue thread.. Is there a way to force the pip install camoco to use pandas-0.23.X as you mention above, instead of pandas-0.24.1?

From the middle of the installation process:

Collecting pandas>=0.19.2 (from camoco) Downloading https://files.pythonhosted.org/packages/e6/de/a0d3defd8f338eaf53ef716e40ef6d6c277c35d50e09b586e170169cdf0d/**pandas-0.24.1**-cp36-cp36m-manylinux1_x86_64.whl (10.1MB)

Thanks! Tyson

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/LinkageIO/Camoco/issues/88#issuecomment-461971300, or mute the thread https://github.com/notifications/unsubscribe-auth/ACcpv71m5uAhGyw_2D0iuz4aizemIRrMks5vLfx0gaJpZM4aey8E .

--

Rob Schaefer, PhD Linkage Analytics, LLC

http://linkage.io http://github.com/LinkageIO

schae234 commented 5 years ago

Hey @TCx64

I updated the version that can be installed via pypi. pip install camoco should work now! Let me know if you run into issues.

Rob

TCx64 commented 5 years ago

Rob,

Thanks for your quick replies! Just wanted to let you know that uninstalling and reinstalling worked this morning! I just had to run "pip uninstall camoco" and then "pip install camoco".

Thanks again! Tyson

Successfully built camoco Installing collected packages: pandas, camoco Found existing installation: pandas 0.24.1 Uninstalling pandas-0.24.1: Successfully uninstalled pandas-0.24.1 Successfully installed camoco-0.6.2 pandas-0.23.4

Hey @TCx64

I updated the version that can be installed via pypi. pip install camoco should work now! Let me know if you run into issues.

Rob

schae234 commented 5 years ago

I'm going to close this. Ill keep in mind during the next big development push to update so that we can work with more recent versions of pandas.