Update to FlexTag 0.2.0

[Horsmann]

• [x] Update to FlexTag 0.2.0
• [x] Provide English & German model
• [x] Add test-cases for models
• [x] Upload models to Zoidberg
• [x] Remove LTL repository
• [x] Include in 1.9.0 release

[reckart]

Do we still have a dependency of FlexTag on TC which in turn depends on an older version of DKPro Core? Wouldn't that be bound to cause trouble for users?

[Horsmann]

Yes. It will always have this dependency. Currently, TC release 0.9.0 uses core 1.8.0.

Wouldn't that be bound to cause trouble for users?

Possible, yes. If this is a concern than we should remove FlexTag entirely from DKPro Core. This situation will not change.

[reckart]

Would it be possible/sensible to include the DKPro Core compatible UIMA components in FlexTag itself?

[Horsmann]

I am not sure what you mean? I already have a class similar to the one in DKPro core already in FlexTag. I could copy the current version over to FlexTag (there have been some changes with the coares tags) and make an updated release. How would this avoid the dependency problem?

[reckart]

The potential problem we currently face is a dependency cycle:

dkpro core (flextag module) -> flextag -> dkpro tc -> dkpro core

If the FlexTag UIMA component would be in the flextag project, then we would not have this cycle:

flextag -> dkpro tc -> dkpro core

Or am I misunderstanding something?

[Horsmann]

You talking about removing FlexTag from the DKPro core development version? As long you have a project in Core this dependency issue would remain, no? core (1.9.0) -> flextag -> tc (0.9.) -> core (1.8.0)

FlexTag can stand-alone and already has a class that does essentially what the current FlexTag-Uima class in Core does (fitting to 1.8.0). I think what you are proposing is already implemented https://github.com/Horsmann/FlexTag/blob/master/flextag-core/src/main/java/de/unidue/ltl/flextag/core/uima/FlexTagUima.java and shipped in the 0.2.0 version.

[reckart]

I was referring to what you said about removing it, yes:

... we should remove FlexTag entirely from DKPro Core.

I didn't know that you already included a UIMA component in FlexTag itself. If that is the case, is the component in DKPro Core then not redundant?

[Horsmann]

Because of its incubation status I keep things redundantly but I would prefer if DKPro would become the reference so that I can drop this annotator from the FlexTag project. For reasons of visibility.

It is certainly not ideal that this dependency issue exist. If it causes problems one cannot use FlexTag then but I think there will be plenty of cases where no issue arises?

[reckart]

I would expect we run into issues when there are certain types of incompatible changes between the latest DKPro Core version and the version that FlexTag/DKPro TC depends on. I assume this would not happen too often and we would probably notice it first in the integration builds - but I am not very sure about this estimation.

[reckart]

@zesch @Horsmann Looking at the dependency hierarchy, I would consider FlexTag to be a product build on top of DKPro Core, DKPro TC, DKPro Lab, etc. With the given dependency footprint and the resulting potential for users getting into trouble, I feel it doesn't make much sense to include it in DKPro Core.

To address the visibility question: would you think contributing FlexTag to DKPro could help increasing visibility? It could e.g. be advertised along the other projects on the overview page, be more likely to be mentioned in presentations, etc.

We don't have to decide now, but with the present dependency footprint, I don't feel particularly comfortable including the DKPro Core FlexTag module in the 1.9.0 release. We can keep it in the master branch for the time being though so it continued to be available in SNAPSHOT builds.

[Horsmann]

I would expect that in the scope of the integration tests issues would probably not arise. At least so far they did not surface. Would it cause problems if it is within the release or only when you try to use it?

[reckart]

Only when people start using it... and then I'm afraid of increased support effort and also leaving a bad experience with users.

We already have cases that require such increased support effort, e.g. with TreeTagger, when people use DKPro Core within Spring-based applications, or when they don't pay attention to the Maven dependency resolution mechanism and get mixed up UIMA/uimaFIT versions, etc.

I see similar problems on the horizon with the DKPro Core FlexTag module - and I'd prefer reducing the number of problematic cases instead of increasing them.

[reckart]

The DKPro Core FlexTag for example transitively depends on the DKPro Lab "simple" UIMA engine - that would cause trouble if one would try to use the DKPro Core FlexTag in a DKPro Lab-based setup that would use another of the DKPro Lab UIMA engines. DKPro Lab was never meant to be a dependency of non-experiment projects.

[reckart]

I think it would be good to get an opinion from @zesch on this.

[reckart]

I don't want to kick the module out by all means. If you want to have it in the release, ok. But I want to make sure that the alternatives have at least been discussed and the concerns/implications/foreseeably problems have been discussed as well.

[Horsmann]

:) I understand your concerns. The various spring versions in the DKPro-layers over TC to Lab certainly call for trouble, too. Hard to say how / if / when issues arise. In some setups certainly.

How about integrating FlexTag in at least one release and see how it goes? If issues arise you can still drop it in the following release?

[Horsmann]

@reckart How do I silence the dependency check?

I have in flextag-core actually all dependencies I need - which are needed if you actually use a model . Not all classifiers and features that are supported are used by the test cases. This leads to many "xxx defined but not used" and "xxx used but not declared" errors which let the build fail.

The dependency check is rather smart and figures out that what it needs are the TC projects. I want Core to use the FlexTag package despite of the analysis saying that there is stuff in it that is (currently) not needed.

[reckart]

See e.g. the OpenNLP module pom.xml.

Please make sure to leave comments in the POM why you force-include certain dependencies.

"xxx used but not declared" should be fixed by adding explicit dependencies on the respective modules. Please take care to choose a proper scope. Things that are only used during unit tests should have the test scope.

[Horsmann]

Thank you. I might have uploaded the models into the wrong repository as it seems. I uploaded them into public-model-releases-local but jenkins looks in ukp-oss-ext-snapshots for them.

How do proceed now? Delete them from model-release and upload them into ext-snapshots?

[ERROR] Failed to execute goal on project de.tudarmstadt.ukp.dkpro.core.flextag-asl: Could not resolve dependencies for project de.tudarmstadt.ukp.dkpro.core:de.tudarmstadt.ukp.dkpro.core.flextag-asl:jar:1.9.0-SNAPSHOT: Failed to collect dependencies at de.tudarmstadt.ukp.dkpro.core:de.tudarmstadt.ukp.dkpro.core.flextag-model-tagger-en-wsj0-18:jar:20170512.1 -> de.tudarmstadt.ukp.dkpro.core:de.tudarmstadt.ukp.dkpro.core.flextag-upstream-tagger-en-wsj0-18:jar:20170512: Failed to read artifact descriptor for de.tudarmstadt.ukp.dkpro.core:de.tudarmstadt.ukp.dkpro.core.flextag-upstream-tagger-en-wsj0-18:jar:20170512: Could not transfer artifact de.tudarmstadt.ukp.dkpro.core:de.tudarmstadt.ukp.dkpro.core.flextag-upstream-tagger-en-wsj0-18:pom:20170512 from/to ukp-oss-ext-snapshots (http://zoidberg.ukp.informatik.tu-darmstadt.de/artifactory/public-ext-snapshots-local/): Failed to transfer file: http://zoidberg.ukp.informatik.tu-darmstadt.de/artifactory/public-ext-snapshots-local/de/tudarmstadt/ukp/dkpro/core/de.tudarmstadt.ukp.dkpro.core.flextag-upstream-tagger-en-wsj0-18/20170512/de.tudarmstadt.ukp.dkpro.core.flextag-upstream-tagger-en-wsj0-18-20170512.pom. Return code is: 409, ReasonPhrase: Conflict. -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:

[reckart]

public-model-releases-local is correct. The message you are seeing is a red herring. Did you run Maven with the -U option?

[Horsmann]

Hm, ok - I need some help. Something went wrong with the deployment as it seems.

The upstream file of the tiger model is not downloadable from the repository. It fails with a resource not found exception. Only the model jar of the tiger model is downloaded. The wsj model/upstream is found, downloaded and can be executed.

I don't see what is wrong? The naming seems correct?

[reckart]

There seem to be files missing in the repo. I wanted to build the models myself and upload them, but the build.xml in the flextag module doesn't seem to be up-to-date.

[Horsmann]

The Build XML is dummy to install Models as an artifact. It does not download the model. Could you delete the tiger model so that I can re-upload it ? The model works if downloaded from the LTL rep - maybe I made a mistake when deploying it to Zoidberg

[reckart]

Can you not overwrite it? Anyway, it looks like the POM for de.tudarmstadt.ukp.dkpro.core:de.tudarmstadt.ukp.dkpro.core.flextag-model-tagger-de-tiger:20170512.1 is missing (at least) - can you deploy that?

[Horsmann]

I tried to overwrite but I get an error when trying that. Probably because I don't have delete permission? I might have forgotten to click on create pom when deploying.

[Horsmann]

Can you just delete the Tiger model from the artifactory? I will try again later and provide other models, too, while I am at it. I will also add a new build.xml

[Horsmann]

Hm, seems like both models have this problem :( - Is it possible to generate the pom for the models afterwards somehow? possible that I forgot to click on generating one.

[Horsmann]

@reckart Could you try again to run the build.xml - the WSJ model should be downloaded and installed. Can you overwrite the version in the artifactory with this one?

[reckart]

The POM must be generated in a special way - namely in the way that the build.xml does it. The POM that Artifactory can automatically create during upload won't work because it does not include a dependency on the upstream model.

[Horsmann]

Oh ok - does the new build.xml create the model in this way? It should?

[reckart]

For the wsj model - yes it should - I am just running it. For the TIGER model not because it's not in the build.xml.

[reckart]

The wsj model is now properly deployed. I'll have to manually delete the previous broken artifacts from the build servers before they work again.

[Horsmann]

@reckart Thank you. I added the tiger model to the build.xml as well. Is there something you have to pay special attention to when deploying the artifacts to Zoidberg? When uploading the artifacts to our artifactory it did work.

Can you deploy the tiger model too? Or shall I give it another try?

[reckart]

Model is deployed. I have also purged the broken models from the build servers, so I expect that builds will be greener again now ;)

[reckart]

Regarding deployment to zoidberg:

• make sure the settings.xml includes the necessary credentials
• set up a special target in the build.xml (e.g. new-models) which calls only the targets for the models you want to deploy - this is particularly important in large build.xml files
• call ant with -Dinstall-artifact-mode=remote
• the models will be deployed into public-ukp-staging-local and have to be moved from there into public-model-releases-local

[reckart]

btw. you can also configure the build.xml to deploy to your own repo. See the description of the configuration properties in ant-macros.xml.

[Horsmann]

Thank you very much. I will add in the near time more models and give it a try.