search

destream-py / destream

A tool & Python 3 library to decompress anything
GNU General Public License v2.0
12 stars 1 forks source link

Another project to maintain #24

Open cecton opened 3 years ago

cecton commented 3 years ago

https://github.com/destream-py/restoredb.py

This one is particularly old. StreamDecompressor was the old name of destream.

If you dont want to maintain it it's okay you can archive the repo. I'm not sure if it is used by anyone at all tbh.

I will move the ownership on pypi to you @jruere and @eumiro . Thanks a lot again for adopting destream ❤️

cecton commented 3 years ago

Geez sorry there is also pgheader which is an optional dependency. I suggest you convert it to a normal dependency (there is no reason for it to be optional imo). Fortunately that one has very little to no maintenance

cecton commented 3 years ago

https://github.com/destream-py/pg_header.py/blob/09e182c524784376dbe748104fe6e3b7bc7908a2/pgheader.py#L4

It's more or less a copy paste of the c code. Since 2014 there are probably new information to retrieve but should still work fine as it is now.

cecton commented 3 years ago

@eumiro can you provide me your username on pypi? So I can add you as owner, thanks!

eumiro commented 3 years ago

@eumiro can you provide me your username on pypi? So I can add you as owner, thanks!

It's the same, eumiro.

jruere commented 3 years ago

Can we archive these, @eumiro ?

eumiro commented 3 years ago

Can we archive these, @eumiro ?

I think you can archive them. Is there actually a way to see the download numbers?

cecton commented 3 years ago

Can we archive these, @eumiro ?

Oh 🥺 You're going to abandon them just like that?

restoredb is a handy script that takes a postgres database compressed in any format and restore it to your postgres server. It's a very nice example on how to extend destream because it creates a few more classes that will handle the different postgres format. (Restoring a postgres dump is not "trivial" as you need to use psql or pgrestore depending on what kind of dump it is, this tool makes it magic).

pgheader is a stupid and simple port of the beginning of the code of restoredb (the postgres binary used to restore database dump in pgdump format). All it does is read the header of the pgdump and return an object with the information about the database (database version, etc...). It's very small to no maintenance at all as thing are supposed to be backward compatible.

But I understand that if you have no experience with postgres it might be a bit difficult to maintain... on the other hand it's an opportunity to learn more! (Learning is awesome imo)

May I ask why you're giving up? @jruere @eumiro

I think you can archive them. Is there actually a way to see the download numbers?

On PyPi you don't need to do anything as thing might break for people who might use it. But on GitHub you can go in the settings and click on the "Archive" button.

eumiro commented 3 years ago

Actually I'd prefer to have a look at these projects later (sorry for accepting the archiving too quickly). I have spent years in postgres and enjoy working with it very much. Let me check the repositories when destream is green.

cecton commented 3 years ago

Thanks :hugs: I'm super happy! :+1:

There is no rush anyway it's been there for years without moving

jruere commented 3 years ago

I glanced over them and they seemed more like examples than production ready tools but it was a very quick look.

I have no problem to leave them there but I don't think I'll do much for them. :shrug:

cecton commented 3 years ago

I was thinking about this and now I remember one of the biggest struggle I had with destream is the examples and real life use cases.

I made it for a company who wanted to let their customers upload database dumps compressed in any format and return another dump with the exact same compressions the customer did. So destream is/was used to decompress the input and then I used pipelines to create the output. (Customers really did use compressions that made no sense together.)

I always thought it was a silly requirement. Then I used destream to make restoredb and this is the only valid use case imo.

I think it would be great to see other projects/real life examples using destream.

jruere commented 3 years ago

Well, the module ships a "zcat" like tool which shows 100% of the functionality, it decompresses whatever you give and outputs a stream.

This is a pretty generic step and can be neatly separated in Unix by using pipes.

Another lovely feature is that decompression is out of process, which is awesome because of CPython's GIL.

The README should mention supported formats and characteristics like this.

cecton commented 3 years ago

I did that? Wait, I didn't know all of that :rofl:

cecton commented 3 years ago

Maybe it should also mention that it fails miserably to actually stream for some files (like ZIP because its header is at THE END of the file).

For those compressions: a temporary file is created (I hope it's not created in /tmp because that would be a big #fail)

cecton commented 2 years ago

Hey everyone :wave:

I hope I didn't discourage anyone on this project... It's been a while I didn't hear from you both and starting to get a bit worried :sweat: I hope I haven't been rude or anything? Anyway, let me know if you need help. I'm alive

jruere commented 2 years ago

Hi @cecton , I'm not discouraged but quite busy. There does not seem to be any problem with the library ATM. It should be updated to newer Pythons but I'm struggling to keep up with actual bugs in other projects. :(

cecton commented 2 years ago

Ah! It's okay no worry. I just had a rough day when I wrote and I was a bit worried for nothing. All good then :+1: