search

visualfabriq / bquery

A query and aggregation framework for Bcolz (W2013-01)
https://www.visualfabriq.com
BSD 3-Clause "New" or "Revised" License
56 stars 11 forks source link

make bquery pip installable #30

Closed ghost closed 9 years ago

ghost commented 9 years ago

Are there any plans to make bquery pip installable? I have the latest bcolz installed. So, to install bquery I just need to clone the repo and run setup.py? Looking forward to running some benchmarks. I noticed in your benchmarks that in some cases it is slower than Pandas.

FrancescElies commented 9 years ago

Hi,

I saw you closed this issue, sorry we did not answer you before, at the moment we don't have much spare time, for us personally would be also very nice to be able to install bquery through pip, but for the moment we don't have any specific plans on that, but for sure would be very nice to have, did you have any problems installing it manually as described in Reame file? Shall we reopen the issue?

Best

ghost commented 9 years ago

Hi, I just saw your message. I can reopen and of course no rush. I have no problem installing the other way and can let you know when I have time again to test it out. Cheers and thanks for your reply:)

Best wishes

CarstVaartjes commented 9 years ago

Great, best to leave it open to remind ourselves to do this. bquery got a change that hopefully makes bquery work with it when it's pip installed, if that's true then we can also make bquery pip installable. In some cases bquery is slower then pandas, but: 1) It's still in the same league while being out-of-core 2) we need to optimize lots of stuff still (see the work that arf1 did on multi-threading for instance; i have some ideas on how to quickly implement that in a way that doesn't collide with the bcolz order-of-things yet still provide significant improvements)

Some of the optimizations can also be applied to Pandas, making Pandas also quicker of course ;)

ghost commented 9 years ago

Hi! I think that when I commented it re-opened. Sorry, been a bit busy recently. Yeah, that multi-threading stuff looks interesting. I'm reading up on threading and non-blocking to understand more. Sounds cool! The best thing about bcolz/bquery is out-of-core. Ideally, I can load my big files into bcolz and then work with bquery and bypass Pandas; Probably will explore a bit more week after next. Cheers.

CarstVaartjes commented 9 years ago

We should try it now that bcolz should have the cython stuff available from pip installs :)

ghost commented 9 years ago

Cool! Thanks for letting me know about the update. Looking forward to playing around with it a bit.

ghost commented 9 years ago

Hi, how's it going:)

CarstVaartjes commented 9 years ago

Hah! Sorry :)

We have some internal discussions around making more than just bquery open source (it is part of something bigger, which includes a scalable services-based architecture and a full html5 front end for importing, reporting and machine learning (encapsulating bquery, pandas, scikit learn etc). Depending on the outcome, hopefully the big release will be made available in december and I think that from october on we will start working towards this. which includes some big improvements in bquery too :) sorry! We are heavy users of bcolz and bquery, so we will definitely maintain and upgrade bquery in any case

ghost commented 9 years ago

Oh, no hurry:) Just curious. Wow, this project you mentioned sounds awesome and uses all the favorite tools. Cheers.

CarstVaartjes commented 9 years ago

hi #52 should have solved this; let me know if it works for you!

ghost commented 9 years ago

Successfully installed bquery! I will close this now:)