eric-kimbrel
commented
9 years ago Its looking good. We are currently doing some testing / bug fixing before we label whats currently in master as the 0.5 release. We can't take anything new into master until we finish that but we anticipate finishing that up this week.
The only thing you are likely to notice when you re-base is that we removed the checked in fig.yml and replaced it with a fig.yml.template to avoid checking in things like local ip's and system paths. Everything else should merge nicely as we are mostly working on UI and extension bugs.
I notice that there is no UI / front-end piece of this yet. Do you plan on adding UI to show an end-to-end feature prior to a pull request or did you just want to get the server in first and go from there? I'd rather see something going from UI to back-end (even if its an early idea to show proof of concept / so we have a way to know things are working when testing) but let us know what you were planning and we can figure it out. We could also do the pull request into a new branch and work on the UI/etc from there. Thats probably the best way to go so we can take some time experiment with different UI ideas and fine tune it.
Last thing i want to mention, when we do add in some UI we need to make sure we include some configuration options that allow us to turn it off / not show it if the user wants (requirement for all new features). We can get that part done but its something for anyone who is working on the UI to keep in mind while they are coding.
I am excited to get this included and super charge the datawake with it!
-Thanks
BurntSushi
We (Diffeo) are hoping to submit a pull request soon that adds a little bit of code to insert feature collections into a
dossier.store. It will also add a newdossier.webDocker container. (This is currently blocked on us open sourcing a few more pieces!)I just wanted to get the process started and incorporate any feedback you might have. (And certainly, we can iterate on the PR too if you want.)
As of now, I have something working on an older copy of Datawake (~1 week old, before your Docker refactor). I'll rebase before submitting a PR, but I figured the essential pieces in Python probably wouldn't change much. Here's our fork: https://github.com/diffeo/Datawake/tree/dossier-integration (See a nice diff here: https://github.com/diffeo/Datawake/compare/dossier-integration)
Here are the highlights:
fig.yml. I'll update this to the new template you have when I rebase.)dossier.storeinserts to yourExtractedDataConnectorclass (which I turned into an abstract base class). I put it there because our DB abstraction librarykvlayerwill work with any supported backend without changing the client code. We don't have an HBase backend yet, so I made sure that the connection is allowed to fail.