Integrate into dbUnit core

[marcphilipp]

Rather than publishing a separate artifact on Maven central, we would like to integrate DataSetBuilder into the core dbUnit project.

A while ago, I have opened a feature request on SourceForge.

[marcphilipp]

In order to do so, we first need to migrate dbUnit to Java 5 and JUnit 4.11.

@opensource21 has created patches for it and attached them to the issue on SourceForge.

[marcphilipp]

@mab has also shown interest to work on getting DataSetBuilder published on Maven central in one way or another.

[marcphilipp]

Suggestion: Let's wait a couple of days for the dbUnit team to respond to @opensource21's patches.

@opensource21: Ok by you?

[opensource21]

Yes it's ok.

[bartoszmajsak]

Hi guys, I also asked dbunit folks about moving all this stuff forward. Let's hope they will be willing to do so.

By the way - I just discovered this nice little library. Looks pretty cool. I will look how I can contribute to it (lemme know if there is something to start right away) and also to incorporate it to Arquillian Persistence Extension.

[opensource21]

So if https://sourceforge.net/p/dbunit/code.git/merge-requests/1/ will be accepted, everything would be fine. I hope they will merge it, so that the honor of your work will still be visible.

[opensource21]

@bartoszmajsak if you want you can improve DBUnit-Validation. I think this is a must have addition to the DataSetBuilder. At the moment it just shows the idea (but have everything I need at the moment). Perhaps you know a better way how to improve the validation.

[bartoszmajsak]

@opensource21 Absolutely! It's pretty cool idea. But this won't happen in May... just to set expectations clear. Too much stuff to do atm. But it's already in my todoist list :) Thanks for the pointer

[marcphilipp]

Great, thanks for your work!

[marcphilipp]

Current state:

1. @opensource21's merge request was rejected.
2. @opensource21 has since resolved the issues that lead to the rejection.

@opensource21 Any news on this?

[opensource21]

Unfortunately I'm not able to start an communication with the DbUnit-Team. There is mailing list dbunit-developer@lists.sourceforge.net where I try to get a contact, with no effort. Without any feedback how they wan't to get documentation and if they are principal willing to integrate it, it's too frustrating. Currently there are 4 open merge-request. It looks that there are only 2 ways to make the situation better:

• Fork the complete DbUnit-Project and integrate the community.
• Create a separate jar.

[favetelinguis]

Any uppdates on the issue? Is it avalaible on Maven, cant find it in dbunit so i guess it is not merged?

[marcphilipp]

No, no updates at all. Feel free to fork it and release it to Maven central. You will need to use a different group id (Sonatype won't allow prefix org.dbunit), though.

[aalmiray]

what about posting releases to bintray.com?

[marcphilipp]

@aalmiray Thanks for your interest in this project. I'm not actively maintaining it since I've stopped using dbUnit.

Regarding bintray.com: I think Sonatype is right to disallow using the group id. Feel free to fork, use a different group id, and publish it there.

[aalmiray]

thanks. I found this project thanks to a team member. The motivation is pretty much the same you had in the beginning: a source code alternative to using XML for defining datasets.

I'd prefer if this feature were to be found inside dbunit core too. Disclaimer: I have had code accepted for dbunit years ago. :smirk:

[marcphilipp]

As I've said: If you have the time to do it, please take the code and see if you can get it into core.

[opensource21]

@aalmiray I tried to integrated it into dbunit. I created a pull request which could be merged directly. After a few month, they create a new version and close the pull request with the comment, it couldn't be merged directly :-/ and they miss the documentation. I asked, where I should document it but never get an answer.

How ever I released it on maven central Group is de.ppi.dbunit.datasetbuilder and version 1.4 I hope this helps some people.

[rafaelvanderlei]

Hey, guys. @marcphilipp @opensource21

I have spent the last couple of days looking for alternatives to DBUnit XML style of preparing data for integration tests when I found this (allow me to say) piece of treasure of yours.

I had worked with DBUnit about 6+ years ago and I remember how I felt a little difficult to maintain all those xml files away from the test code (hard times to update all dataset xml files when new non-nullable columns got added or eventually when some columns got renamed).

After a long while away from DBUnit, I'm about to go back to working with integration tests again and I want to reduce the painful experience from the past to the minimum.

I first found a project called DBSetup (http://dbsetup.ninja-squad.com/index.html) and I was happy to see some effort on a way to prepare the data writing java code. It is not a bad solution imho, but to be productive on using DBSetup to write data preparation code that is easy to maintain along with changes in the database schema, there would need a lot of improvements on top of it. Some of them are already implemented in DBUnit (regarding to the things one can do once has loaded an IDataSet, like filtering, replacing, asserting, database operations, etc.). Other features required would be things like the ability to declare table metadata only once, in a type-safe fashion, allow default values to be applied, etc.

So the ideal solution to my thoughts at that point would be an upgraded version of DBSetup that could also be integrated with DBUnit.

Fortunately, out of a sudden I ran into the merge request that @opensource21 had opened in DBUnit issue tracker and that's when I found out about dbunit-datasetbuilder. The custom BasicDataRowBuilder implementation per Table allows one to isolate metadata details into one place , apply default values and to build datasets in a type-safe fashion. To go one step further it comes up with a code generator that takes a IDataSet (that can be loaded by any of the means supported by DBUnit) and creates both the code for the BasicDataRowBuilder and even the code that uses it to generate the given IDataSet. It would be nice if some features from DBSetup were presente here, (actually, the only things from there that would be worth bringing to here would be the support for value generators and the ability to generate repeated rows) and I'll definitely put some thoughts into contributing to this project.

By the way, I can't understand why this contribution hasn't been accepted yet to make it into DBUnit Core. I've seen the history of the merge request and I believe this is so ready to go... it works, it has test code, and it has very nice documentation... it would be a great addition to DBUnit.

Congratulations and thanks for sharing this. Sorry for the long post.

Regards, Rafael.

[marcphilipp]

@rafaelvs Thank you for your kind words! I'm glad you like it.

Since dbUnit doesn't seem exactly eager to integrate this I think the only way forward is that somebody publishes this code as a separate artifact. Since I'm not using it at the moment and working on JUnit is keeping me busy, I'm really looking for someone to take over. I'd be happy to rename/-locate this repository.

@rafaelvs Please let me know if you're interested.

[rafaelvanderlei]

@marcphilipp Well, I am definitely interested to extend dbunit-datasetbuilder to include at least those features I told before (use of value generators and allow to repeatedly add rows). Also, I was thinking to suggest a few changes in the code, like package organization and perhaps move CustomRowBuilderGenerator to another module/artifact (like *-tools).

As of distribution, I also have the intention of contacting dbunit team to understand what is missing for them to accept this code. If they happen to reject or to demonstrate it would be somehow difficult to accept further changes to this code, I would pretty much prefer to publish as a separate artifact, as you suggested so it would be easier to upgrade. (in this case, I think DBUnit team should, at the very least, mention this solution in their documentations as a known 'unofficial' alternative to build IDataSet's)

As of code repository name or location, I honestly don't know much about the options or implications, since I don't have much experience (close to zero) on contributing to open source projects. If I understand you correctly, you won't have the time to manage this repository (to accept pull requests, release new versions, publish to maven central, etc.) and so you're willing to move it forward?

I was thinking on forking this repository to start contributing and send it back to you, but if you won't be able to manage pull requests, please let me know which would be the best way to make my further contributions to get back to the community.

[marcphilipp]

Feel free to fork this code, of course!

Maybe you can join forces with the other participants in this issue and start a GitHub organization?

[opensource21]

@rafaelvs I would recommend to try to get in contact with the DBUnit-Maintainers. Perhaps you have more luck than I. It's definitely the best way. Second I would recommend to start to fork myversion it has small improvements. You can look at dbunit-validation too.

About a separate module for the generators: I wouldn't do this, because the generator must match exactly to the code. So separate versions can make trouble and will create tickets.

About publish in maven central: I have released it and can do it again if there will be a new version necessary.

If there are more person who want develop I think we can easily transfer the project to an organization. But I think we should first start with more development and do then the restructuring. Try it first and see if you really glad with the solution.

About the state of dbunit-validation: I implement what I need in one private project. At the moment I would be glad if I had a real database in my project.

[rafaelvanderlei]

@opensource21 the idea of having the generators in a separate module is because it's not needed at runtime. It's just a tool to support coding the dataset builders. By separate module, I mean a module in maven "brother" to the dataset builder, both having the same parent. This way, I believe the chances of mismatching the code generators from the code to be generated won't be much of a problem, would it?

What improvements did you make in your version of this library? Is it easy to list? I made some coding this weekend, just to play and get more used to the code, and it came out that I redesigned the library quite a bit.

For instance, instead of having the DataSetBuilder implement IDataSetManipulator that builds a CachedDataSet, I implemented a new class, ListDataSet (working on to find a better name), that extends CachedDataSet and a ListDataSetProducer that implements IDataSetProducer to create the ListDataSet from a java.util.List of Table, a new class that contains a List of Row, that contains a Map<String,Object > for <columnName, columnValue)... With that design, we actually need a API to build the List of Tables and Rows (simple POJOs), instead of the DataSet itself.. and then we pass this List to the ListDataSetProducer that will be used to create the ListDataSet. (It could be any Collection, but I chose to use a List among others just to ensure the Tables will be displayed in the DataSet in the same order they were added to the Builder).

I'll try to commit the changes that I propose in my repository asap, but it may take a few days (at most by next sunday)

By the way, I had this idea by looking at other DataSet implementations (from yaml, json sources, etc.) that I found in @bartoszmajsak 's arquillian-persistence-extension ... @bartoszmajsak , I plan to make some work on that project too as I intend to use it to write integration tests for a new project to come, especially I'd be glad to integrate this "java-builder-approach" to provide IDataSet's. I plan to think of ways to provide the dataset builder using the annotations @UsingDataSet and @ShouldMatchDataSet (maybe include new attributes like type or builder-method .. or once informed type=builder, we can assume the builder-method will use some pattern involving the test method name... I don`t know.. I haven't dug into the code enough to suggest the best approach)

@opensource21 As of dbunit-validation, I cloned and had a look at the code... perhaps I didn't have the same vision as you, but so far I couldn't see a real world use for it as I think we should assert for the exact value for a better verification that everything is actually working as expected.

[opensource21]

@rafaelvs Of course you could put it into different jars, but I doesn't think that it is so important to have such a clean path for test. Even if you release both with the same version number it makes it easier to have different versions in one project. Every time I used it I put the Generator to the tests, it doesn't matter at runtime.

About the idea of ListDataSet: I can't see the advantage of this approach. The Builder-Pattern looks perfect to me. To create a List first and then transform them to a DataSet looks to me only more complicated. So what is the benefit you want to achieve with it? But perhaps it's better you create a separate ticket for it. Discussion to many issues in one ticket is a bad idea. @marcphilipp What do you think about the idea?

About improvements: See https://github.com/opensource21/dbunit-datasetbuilder/commits/master the last commits. Better Generator, prepare for DBUnit and Maven Central.

About Validator. You can see real live tests for SZE which use the validator. For example: Is it really necessary to check that the version is increased by one? I think if you want to have good maintenance you should make if more flexible. Or at FuWeStaSample you can see how to check if an id (which create via hibernate) is greater 0 (that's enough for me) and stored the value in a variable so that you can reuse it later. I found the idea at checkerberry a German commercial testing tool. I hope this makes the value more clear.

[rafaelvanderlei]

I agree it's better to create new tickets to discuss differente tickets, so I'll be brief here and later I'll create separate tickets.

I got what you mean about the generator. I agree to keep it together with the main library as it would be too much effort and error prone for too litte benefit. Maybe just a better package organization would be enough.

The idea of the ListDataSet is actually a refactory to make it easy for us to maintain the DataSetBuilder code itself. For the DataSetBuilder users, nothing (or little) would change.. the end user of the library would still create the DataSet using the Builder Patter, which is perfect approach to me too. The purpose od the ListDataSet and the ListDataSerProducer is to be used internally by the code of the DataSetBuilder.. I think it would be cleaner and less complex to maintain than it is in its current state.

I'll have a look at the improvements and validator examples. Thanks for the pointers.

[rafaelvanderlei]

@opensource21 I forked your repo and created new issues in my fork. If you could provide some feedback, I'd be grateful.

Unfortunately, I haven't commited the changes I made. I think I'm almost finished (already implemented the most important issues - 1 to 3). Just need to work on some clean up, few testing and documenting, but I guess I'll be able to commit everything by the end of the day.

[rafaelvanderlei]

@opensource21 @marcphilipp I finally had the time to commit. If you're interested, please have a look at my repo for the code changes and provide some feedback. There is some documentation written in the issues #1, #2 and #3 that might be useful to understand what has changed. Unfortunately some public builder api's changed a little, so I thought of updating version number to 2.0.0 .. if necessary, we could work on migration guide and/or writing some code adapters, so that code written with previous versions doesn't need to change.

@bartoszmajsak after finishing the code cleanup, testing and documenting, I would like to see if we can integrate this to Arquillian Persistence Extension. Are you still interested?

[marcphilipp]

I'm quite busy with JUnit at the moment. But I will take a look this weekend. Sorry for the delay!

[marcphilipp]

Over the weekend I've come to realize that I won't have time to review your changes. Nevertheless, it's great to see some progress here – keep up the good work!

Regarding this issue: Would you mind continuing the discussion in one or multiple new issues over at the new repository so I can close it here?

[rafaelvanderlei]

@marcphilipp I understand your lack of time. Don't worry about it.

Regarding this issue, is it possible to migrate the discussion to a new issue in my repo? Or must I create a new issue there and then you close this one referencing the new issue?

[marcphilipp]

You'll have to create a new issue and link it to this one using this syntax: marcphilipp/dbunit-datasetbuilder#6.

[rafaelvanderlei]

I created the issue. Thanks, @marcphilipp !

[marcphilipp]

Thanks, @rafaelvs!

Discussion will continue in rafaelvs/dbunit-datasetbuilder#10.