bethard
commented
9 years ago Comment #1 originally posted by ClearTK on 2010-12-30T21:00:38.000Z:
I have made good progress on this issue this week. I have a first pass of the toolkit project reorg complete now. All the projects are in place, the code is compiling, and the tests are passing. I still have a laundry list of things I need to take care of before I merge this branch back into trunk. So, now is a good time to take a look at this branch of give me some feedback. There will still be opportunities to rearrange things after its moved back to trunk too. I would like to do the merge and create a new cleartk release next week.
Original issue 160 created by ClearTK on 2010-09-09T15:18:28.000Z:
Philip:
I would like to propose a pretty significant change to the way our code is organized. Here are my motivations:
In yesterday's email I gave a quick summary of how I would like to see the framework project split up (see below). I think this will go a long way towards easing peoples fears of open source license entanglements as well is improve our ability to communicate what the implications are of using the various wrappers. This change seems like a slam dunk to me - so I won't spend time justifying why we need to do this unless objections are raised.
Similarly, I would like to see the ClearTK toolkit project split up into meaningful subprojects. I have a use case where I would like to use just the tokenizer and part-of-speech tagger in ClearTK. Should I really have to import all of ClearTK along with the mountain of dependencies to do this? no. Here's a sketch of how the projects might be split up:
-ClearTK-temporal
I think some of the code that is currently in the toolkit project should not survive the restructuring. The code in the ne package is crap and barely does anything, for example. All of the OpenNLP wrappers need to disappear or go into a separate project(s). Certainly, we could write our own sentence segmenter. The parser wrapper could be put in a subproject that depends on ClearTK-syntax. I have a wrapper for the Berkeley Parser - but it is GPL and needs to be a separate project that also depends on ClearTK-syntax. Jinho has a dependency parser that will almost certainly get a ClearTK/uima wrapper - if not be refactored to actually use ClearTK. So, ClearTK-dependency-parsing would be a logical subproject.
So, there are a lot of details I've glossed over. But does the rough sketch make sense?
Philipp:
we talked about this before, so you know that I basically agree with you. I think splitting up the framework by dependencies (especially on the ML libraries) is the obvious thing to do. I'm not entirely sure how the toolkit should be split up. On the one hand, I think we also want to isolate major dependencies from each other where we can, but it also makes sense to group things by task. I'm not entirely convinced by your proposed split below; for example, it would seem that the CoNLL'05 collection reader should then go into "corpora", but it could just as well go into "semantic roles", because it's so tightly linked to that task. And if we group collection readers in "corpora", why is there another set of collection readers in "util"?
As an alternative / modification, how about a "collection readers" package, which contains only the generic collection readers (like FileSystem), which are general purpose and don't depend on any specific types; separate "tokens", "semantic roles", "syntax", "temporal", ... packages for the separate tasks, each of which may contain type system additions, and collection readers that are either specific to the task (e.g CoNLL'05), or they require the task's type system to function (e.g. Treebank in "syntax"), but these packages only contain ClearTK implementations; and then packages for each of the major dependencies (such as OpenNLP), which may contain components for multiple tasks (and thus may depend on the task-specific ClearTK packages).
Philip: There are two ways that dependencies can be a pain - license incompatibilities and version incompatibilities. If I download a package that is dependent on the universe and all of its dependencies are out-of-date with ones I have already added to my package - this is really annoying. But this is less of a show-stopper than having licensing issues with dependencies. I want it to be easy for people to download and use ClearTK without worrying about dependencies with other licenses. Code that depends on an LGPL library for example, will have to be moved into its own library. However, if we simply structure sub-projects based on dependencies we will end up with something rather unintuitive and difficult to navigate I think. So, there will be some art to how we restructure everything and we will just have to discuss it when we get started on it.