search

frictionlessdata / frictionlessdata.io

The main repository of the Frictionless Data project. Website, issues, and discussions
http://frictionlessdata.io
MIT License
141 stars 53 forks source link

Maintenance of deprecated libraries #695

Closed pwalsh closed 3 years ago

pwalsh commented 3 years ago

@akariv and I would like to take over maintenance of several codebases that are deprecated by the newer approach of bundling everything into the Frictionless library.

Given that these libraries (i) have been renamed on GitHub (now differ from their names on PyPI), and (ii) are declared as not receiving any further updates, we'd like to move them to a different organization (unless the policy for naming repos under the frictionless org changes). We definitely want to update these libraries and therefore remove deprecation notices, etc.

The libraries we are interested in have @akariv , or myself, as primary committers or as part of the original design team of the library.

We are interested in maintaining:

There may be more but these are libraries we actively use that are either marked as deprecated or essentially abandoned by the core team supporting Frictionless Data. We use them and want them to continue to grow.

roll commented 3 years ago

@pwalsh Hi Paul! It sounds really great.

Would you be happy to keep the projects in this organization on Github if it's renamed back to the initial names (tableschema-*) and all the deprecation notices are removed?

Currently, we consider this organization more like a developers hub or incubator when each project has its own LEAD, for example, https://github.com/frictionlessdata/tableschema-java/blob/main/LEAD.md. The lead makes all tech decisions and has the last call on them. Basically, Open Knowledge doesn't really meddle into the development except for providing some help like https://github.com/frictionlessdata/datapackage-php/issues/50. For example, Dapotian does similar to what you said regarding JavaScript libraries: tableschema/datapackage/frictionless. Open Knowledge is not a part of this development.

We, of course, would like to keep tableschema/datapackage libraries as a part of the frictionless toolkit - https://libraries.frictionlessdata.io/docs/status - note that almost all the listed projects there are not directly maintained by Open Knowledge.

Also, currently, @lwinfree works on some governance concepts so hopefully we will be able to institutionalize the written above soon, so, for example, LEADs will get their guarantees written down as a part of Frictionless Agreement or something.

PS. BTW do you know what's the status of Data Package Pipelines?

pwalsh commented 3 years ago

Would you be happy to keep the projects in this organization on Github if it's renamed back to the initial names (tableschema-*) and all the deprecation notices are removed?

Sure. Those are the issues to work around, so if they can be handled here then that is great.

We, of course, would like to keep tableschema/datapackage libraries as a part of the frictionless toolkit - https://libraries.frictionlessdata.io/docs/status - note that almost all the listed projects there are not directly maintained by Open Knowledge.

Sure. I just figured that as they are marked as deprecated/bugfix only, and they are reimplemented (not depended on) in Frictionless, the APIs will likely diverge but that is ok from our perspective.

BTW do you know what's the status of Data Package Pipelines?

Need @akariv to respond on that. I am not currently using it.

lwinfree commented 3 years ago

Hi @pwalsh!

I'll mostly reiterate what Evgeny has said - it would be awesome to have you & Adam maintain those libraries. We're working on reimagining governance for the project right now, trying to involve community members more. This would include defining the roles for maintainers in a more clear way. Right now, maintainers don't have strict agreed-upon tasks, except that they will "maintain" the code. We'd like to formalise that some, so maybe we can test that out with you two!

I agree with Evgeny that the repos should stay under the Frictionless organization, and am happy with renaming them/removing the deprecation warnings as Evgeny suggested.

The other thing is that we'd really like all users to start using Frictionless-py instead of the older Python code. Could your work/projects migrate to Frictionless-py? Are there features missing in Frictionless-py that would make migration easier/more desirable? Would y'all be interested in having a call to discuss that?

Hope you're staying well & it's nice to hear from you :-)

pwalsh commented 3 years ago

This would include defining the roles for maintainers in a more clear way. Right now, maintainers don't have strict agreed-upon tasks, except that they will "maintain" the code. We'd like to formalise that some, so maybe we can test that out with you two!

We can talk about it when it is relevant. Not clear what "strict agreed-upon tasks" might mean :).

The other thing is that we'd really like all users to start using Frictionless-py instead of the older Python code. Could your work/projects migrate to Frictionless-py? Are there features missing in Frictionless-py that would make migration easier/more desirable?

We have a bunch of data products in production running workflows using Dataflows, which directly depends on the libraries mentioned and a couple more. Swapping out dependencies is a big deal as data pipelines etc are maintained and evolve over time.

The new Frictionless library looks good but it would require quite a bit of work to fully assess, integrate, test, as it is essentially a rewrite of the entire Frictionless Data software stack; compared to existing libraries that are already widely used, it looks like a heavier lift than just iteratively updating libraries we already know. Also, Frictionless bakes in an ETL framework and we use another (Frictionless Data-based) ETL framework, and not sure if/how that impacts anything.

Bottom line - we will check out Frictionless but, the frictionless ( :) ) approach for us is to iteratively enhance/maintain the libraries we already use. Better the devil you know and all that.

Hope you're staying well & it's nice to hear from you :-)

Thanks Lilly - hope you are well too. Great to chat and I'll just on a community call sometime to say hello 👋

roll commented 3 years ago

Hi @pwalsh,

Do I understand correctly that you and Adam will be working on the libs on behalf of While True Industries? I just need to grant permissions to the libs and can use this entity as your team or can do it for you and Adam as individuals.

pwalsh commented 3 years ago

Hey @roll

@akariv and I do data products and services as whiletrue, yes, but we both, especially Adam, do a bunch of other stuff as well with other entities. It can be as a team, but also as Adam and I are individual contributors to frictionless data in general maybe it makes sense to just add as both individually (as well)?

Whatever works for you and how you manage the org here on GitHub is AOK.

roll commented 3 years ago

Hi @pwalsh @akariv,

I believe that all is set up now.

I would suggest that we use separate issues in the repos if something doesn't work e.g. I didn't grant access to some repo or we need to add you on PyPi etc

Thanks for supporting the FD libs!