There should be a command line interface

[GoogleCodeExporter]

Hello,
IMHO there should be a command line interface to tubular. I have started to 
create on based on the Apache Commons CLI Project to learn the Tubular API. I 
will attach the code to this issue when it's somewhat stable and you are opel 
to integrate a command line interface.

Regards
     Torsten

Original issue reported on code.google.com by torstenm...@gmail.com on 30 Oct 2010 at 3:59

[GoogleCodeExporter]

You're right, the project needs a command-line interface. I'll add a Maven 
module with basic support for command-line execution so that I can integrate 
your changes.

Original comment by herve.qu...@gmail.com on 30 Oct 2010 at 10:03

[GoogleCodeExporter]

I just added an empty class: CommandLineExecutor.
I also updated the Maven POM so that a shaded JAR (with all dependencies) is 
built along the standard package phase.

To test the command-line interface:

$ cd tubular-core
$ mvn package -Dmaven.test.skip=true
$ java -jar target/tubular-core-0.1-SNAPSHOT-cli.jar

Original comment by herve.qu...@gmail.com on 30 Oct 2010 at 11:02

[GoogleCodeExporter]

Thanks for the fast answer and work on it. I will take a look on it on Monday, 
as I have no access to the notebook with those files on the weekend.

Original comment by torstenm...@gmail.com on 31 Oct 2010 at 4:56

[GoogleCodeExporter]

Please find attached a diff file containing a first implementation.
Unfortunately an exception is thrown in "new Configuration()". Do you have an 
idea where the problem could be?
Exception in thread "main" java.lang.ExceptionInInitializerError
        at org.trancecode.xproc.Configuration.getDefaultStepProcessors(Configuration.java:161)
        at org.trancecode.xproc.Configuration.<clinit>(Configuration.java:71)
        at org.trancecode.xproc.cli.CommandLineExecutor.main(CommandLineExecutor.java:80)
Caused by: java.util.NoSuchElementException
        at java.util.ServiceLoader$LazyIterator.next(Unknown Source)
        at java.util.ServiceLoader$1.next(Unknown Source)
        at com.google.common.collect.Iterators.getOnlyElement(Iterators.java:275)
        at org.trancecode.logging.spi.LoggerManager.getLoggerManager(LoggerManager.java:34)
        at org.trancecode.logging.Logger.getLogger(Logger.java:38)
        at org.trancecode.xproc.step.AbstractStepProcessor.<clinit>(AbstractStepProcessor.java:31)
        ... 3 more

Original comment by torstenm...@gmail.com on 1 Nov 2010 at 10:22

Attachments:

• issue4_1.diff

[GoogleCodeExporter]

Thanks for your contribution. I like the command-line syntax you defined.

I had to move the class to a separate module, though. Indeed, some dependencies 
in tubular-core are not mandatory (scope "test" or "provided" in Maven, e.g. 
log4j) and they were not included in the uber-JAR. This is why you encountered 
the error you reported. We can now define dependencies that are specific to the 
command-line module (e.g. commons-cli).

The new module is tubular-cli. It is built along the other modules. To build 
and run manually, same as before:

$ cd tubular-cli
$ mvn package -Dmaven.test.skip=true
$ java -jar target/tubular-cli-0.1-SNAPSHOT.jar 

BTW, I also added a more relevant error message in the logging API so that we 
can better figure out what is wrong when such error occurs in the future.

Original comment by herve.qu...@gmail.com on 1 Nov 2010 at 1:15

[GoogleCodeExporter]

I'm sorry, I just realized I forgot to give you credit in the commit message. 
Anyway, the Javadoc @author tag references your name.

I added a wiki page with the list of contributors just in case:
http://code.google.com/p/tubular/wiki/Contributors

Original comment by herve.qu...@gmail.com on 1 Nov 2010 at 1:25

[GoogleCodeExporter]

Thanks for the fast response, much better than with many commercial products.
The command line interface has been inspired by the ones of Saxon and AltovaXML.
It's now working at least basically. I will do some testing and eventually 
bugfixing with the CLI and give updates in this issue.

Original comment by torstenm...@gmail.com on 1 Nov 2010 at 3:00

[GoogleCodeExporter]

FYI, I encountered the following error:

$ java -jar target/tubular-cli-0.1-SNAPSHOT.jar -x something.xpl
Exception in thread "main" java.lang.NullPointerException
    at org.trancecode.xproc.cli.CommandLineExecutor.main(CommandLineExecutor.java:88)

I just commited a fix for that. Basically the method Option.getValues() returns 
null when no value has been set. But Java does not allow us to iterate over a 
null value. As a workaround I added some utility method that returns an empty 
array instead of null.

Original comment by herve.qu...@gmail.com on 1 Nov 2010 at 4:49

[GoogleCodeExporter]

Thanks for the information, but I think we both fixed the same bug. I used the 
Option API the wrong way. Parsed Option's have to be accessed using the 
CommandLine API. I will attach a new file tomorrow. I also changed the URI/ 
File parsing to support both file's and URLs to allow pathes containing 
backslashes on Windows.

Original comment by torstenm...@gmail.com on 1 Nov 2010 at 6:45

[GoogleCodeExporter]

Like announced, please find attached my changes as diff and exported bundle so 
you can either just unbundle the bundle or apply the patch to your tree.

Could you give me commit rights to an own branch so we have a more dynamic 
exchange point? It would allow me to publish my changes more regular without 
publishing every small change as a diff or bundle file here.

Original comment by torstenm...@gmail.com on 2 Nov 2010 at 9:25

[GoogleCodeExporter]

Forgot diff and bundle.

Original comment by torstenm...@gmail.com on 2 Nov 2010 at 9:26

Attachments:

• issue4_2.diff
• tubular-bundle-issue-4-1.bundle

[GoogleCodeExporter]

I have successfully used the CLI to process a pipeline (at least I guess so). 
My problem now is, that I do not get an output. What should normally be used to 
set the default input and output to the standard console? I think I can bind 
the primary input port to a StreamSource(System.in) or the like, but how would 
I do it for the output?

Thanks in Advance
     Torsten Knodt

Original comment by torstenm...@gmail.com on 2 Nov 2010 at 3:55

Attachments:

• tubular-bundle-issue-4-2.bundle

[GoogleCodeExporter]

Thanks for your patches.

Regarding the commit rights, I think the best way is to build your own clone of 
the source tree. This is what someone else (Romain Deltour) did earlier to 
provide some patches:

http://code.google.com/p/tubular/source/clones

Anyone is allowed to clone any repository I believe. And it makes merging back 
and forth easier. If you clone the repository, I can provide you with a 
separate job on the continuous integration server, just as for Romain's 
development branch:

http://www.trancecode.org/hudson/job/rdeltour-tubular/

Unfortunately I won't have time to review and commit your changes tonight. But 
hopefully I will be able to work on Tubular tomorrow evening.

Regarding your latest question, I think I would do the following:

- Find out the primary port with 
PipelineResult.getPipeline().getPrimaryOutputPort()
- Retrieve a single XdmNode from the port using PipelineResult.readNodes()
- Use XdmNode.toString() [1] to print the result node to System.out

But maybe we need something easier to use, such as the following method:

PipelineResult.writeNodes(PortReference port, Result result)

[1] 
http://www.saxonica.com/documentation/javadoc/net/sf/saxon/s9api/XdmNode.html#to
String()

Original comment by herve.qu...@gmail.com on 2 Nov 2010 at 10:32

[GoogleCodeExporter]

I noticed you created your own clone and I've been able to review the code in 
your clone. Here are a few comments:

- Great job at using the same conventions and formatting as the rest of the 
project. It will make merging easier.
- I think we should rely on the URIResolver from 
PipelineProcessor.context().getUriResolver() to retrieve any Source object. I 
was about to commit some code in this regard before I noticed your changes. 
I'll try and merge the logic with your code. The main benefit here would be 
that the CLI would support URIs, not just URLs so that a custom resolver (with 
a catalog) will fit nicely. It also reduces the amount of logic in the CLI code 
as the URI parsing and resolving logic is delegated to the URIResolver. The 
only thing left IMHO would be the Windows path format support.
- The same goes for the output ports: we could use the OutputURIResolver from 
the Saxon Processor.
- The pipeline output port cannot be bound to a Source. You'll have to retrieve 
each node from the output ports using PipelineResult.readNodes() and serialize 
them to the right location using a Saxon serializer for example. Here we could 
use the above mentioned OutputURIResolver to delegate any URI parsing or 
resolving logic.
- You are using RunnablePipeline.withOption() to set a parameter while there is 
actually a method named RunnablePipeline.withParam() for this purpose.

Thanks again for contributing to the project. I'll try and spend some time 
tomorrow merging your changes to the trunk.

Original comment by herve.qu...@gmail.com on 3 Nov 2010 at 11:56

[GoogleCodeExporter]

- No problem. I was on Java Style Guide defaults in NetBeans and later noticed 
that you seem to be nearer to the usual C/C++ style.
- Agreed. I don't know why I didn't use it, especially because the code 
completion showed my the getUriResolver method. I have implemented this now and 
also support file names using getUriResolver((new 
File(optionValue)).toURI().toString(), null).
- For the usage of the OutputURIResolver I have created a code in a comment 
because I have to check how to get the Data from the XdmNode into the result 
first. I think in the future the tubular API should provide a way to specify 
the output/ result more easy, e.g. by providing an API call accepting Result 
objects.
- Already fixed, but didn't commit yesterday.
- Thanks, copy'n'paste bug.

Pull from my repository before to get the latest changes before.

Original comment by torstenm...@gmail.com on 4 Nov 2010 at 8:50

[GoogleCodeExporter]

I had quite some trouble merging your branch. I guess I need a bit of practice. 
Also, some of your files were using the DOS file format and it messed up the 
diff/merge mechanism.

While I was there, I applied a few improvements to the class. You can read 
about them in the commit logs but here are the most relevant ones:

- Explicit Log4j setup, geared toward end-users (no category nor logger level 
display)
- Allow the method that resolves Source objects to throw exceptions (with the 
error message passed as an argument to the method)
- New options: --verbose and --version
- I sorted the option declarations and regrouped option declaration and setup
- I also added some comments regarding no-op code (library parsing)

I believe I'll have some time to work on Tubular this weekend. I'll try and 
have a closer look at the following issues:
- output port binding
- library parsing and import

Original comment by herve.qu...@gmail.com on 5 Nov 2010 at 12:01

[GoogleCodeExporter]

It seems me to. I have activated Mercurial's EOL extension now to take care of 
this. The configuration is in file .hgeol in my repository.

Great work and it would be really good if you would find time to take a look at 
the output port binding and library parsing and import. I will work on 
automated tests next. First changes are in my repository, but currently failing.

Original comment by torstenm...@gmail.com on 5 Nov 2010 at 8:03

[GoogleCodeExporter]

I have pushed the first working (basic) test into my repository. I also enabled 
the check style plug-in but only have a basic configuration in so far.

Original comment by torstenm...@gmail.com on 5 Nov 2010 at 8:58

[GoogleCodeExporter]

I pulled some changes and applied some improvements to the main class.

I removed the main_internal() method. The class now relies on an instance to 
parse the arguments and run the pipeline. All inputs and outputs (e.g. stdout) 
are now passed as arguments to the execute() method in order to ease unit 
testing. We should be able to test the command line interface without spawning 
a new process now.

Little advice: please don't forget to format XML files using 2-spaces indents.

I'll have a look at the output port binding issue now.

Original comment by herve.qu...@gmail.com on 5 Nov 2010 at 11:06

[GoogleCodeExporter]

I just added a test method for the command line interface.
In each child directory, the test method will parse test.properties to find out 
about arguments, expected exit code (optional), file to use as stdin 
(optional), and a file that contains the expected stdout content (optional).

So far, it is failing because of the output port binding issue in 
CommandLineExecutor. I'll try and fix that.

Original comment by herve.qu...@gmail.com on 6 Nov 2010 at 12:17

[GoogleCodeExporter]

I removed the primary input and output options and fixed the stdin and stdout 
support. The test now passes. Read the commit logs for more details.

TODO: proper file-backed output port binding

Original comment by herve.qu...@gmail.com on 6 Nov 2010 at 12:37

[GoogleCodeExporter]

I discovered a flaw with this new approach. Indeed, when there is no content in 
stdin, the pipeline processor will hang forever. This is due to the current 
architecture of Tubular so I decided to remove stdin support entirely. Again, 
some more info in the commit logs.

By the way, I also fixed some misnamed argument that was causing port binding 
to fail anway:

final Properties portBindingProperties = 
commandLine.getOptionProperties(optionOption.getOpt());

should have been:

final Properties portBindingProperties = 
commandLine.getOptionProperties(portBindingOption.getOpt());

Original comment by herve.qu...@gmail.com on 6 Nov 2010 at 1:06

[GoogleCodeExporter]

- Very good way to get rid of spawning a java process and still being able to 
test standard system streams.
- Sorry, I normally try to take care of the indentation. The problem is that 
NetBeans doesn't allow me to specify a different indentation level for XML than 
for Java or at least I haven't found a way to specify it.

BTW, you shouldn't try to merge changes from me currently.
I have started to generate an .EXE file using Launch4J and will also add an 
Microsoft Installer (.MSI) File so it feels like a normally installable and 
executable Windows program.

Original comment by torstenm...@gmail.com on 8 Nov 2010 at 9:09

[GoogleCodeExporter]

Torsten,

Are you still interested in developing this feature or shall I takeover the 
development? I have been working on a new CLI library based on reflection and 
annotations as a side project. I might use it in Tubular someday...

Herve

Original comment by herve.qu...@gmail.com on 6 Jan 2011 at 9:51

[GoogleCodeExporter]

Hello,

I created a clone and done little works:
- migrating to saxon 9.3
- changing comparaison method with xmlunit in TcAssert class
- implementing ComparePipeline usage in tests
- correcting StoreStepProcessor for access to file scheme => the p:store xproc 
test is now ok

I'm interested to contribute to this great project by implementing unsupported 
steps.  

Emmanuel

Original comment by emmanuel...@gmail.com on 3 Feb 2011 at 4:06

[GoogleCodeExporter]

Hi Emmanuel,

Thank you for your contribution. I am currently in the process of reviewing and 
testing your submitted changes. It looks very interesting so far.

BTW, I am not sure this is the right place to announce your contribution as the 
ticket was about contributions from another person. Maybe you could create 
another ticket to track your changes. Or better yet: I just added two new 
mailing lists for the project among which is a developer mailing list.

I'll most probably merge your changes and give you some feedback during the 
weekend.

Thanks again.

Herve

Original comment by herve.qu...@gmail.com on 5 Feb 2011 at 10:48

[GoogleCodeExporter]

The feature (command-line interface) has been implemented. I am closing this 
ticket.

Original comment by herve.qu...@gmail.com on 8 Feb 2011 at 11:21

• Changed state: Done