Adding commits and branches to DID-matlab

[stevevanhooser]

The brainstorm for the SQL implementation is to have a meta-table for each database document, with a column CREATED and another column DELETED. The branch name where the document is created is listed in the document's CREATED entry, and any branch where the document is deleted is listed in the DELETED column. Then, if one is on branch X, one searches to make sure that the CREATED field is contained in X, and that none of the entries in the DELETED column contain X.

[altmany]

I just uploaded a commit with numerous fixes to both did.database and did.implementations.sqlitedb - please test it carefully, starting with the public methods of did.database. Here's a sample session:

db = did.implementations.sqlitedb('test2.sqlite');
db.add_branch('b5')
db.add_branch('b5a')
db.add_branch('b5b','b5')
db.add_branch('b5b2','b5b')
db.display_branches('b5')
branch_ids = db.get_sub_branches('b5')
db.set_branch('b5')
branch_id = db.get_branch
branch_id = db.get_parent_branch('b5b')
db.add_doc(doc)
db.add_doc(doc,'b5')
db.add_doc(doc,'b5a')
db.add_doc(doc,'b5b')
doc_ids = db.get_doc_ids()
doc_ids = db.get_doc_ids('b5a')
query = did.query('speedtuning_calc.depends_on.name','exact_string','stimulus_tuningcurve_ID','').or(did.query('speed_tuning.tuning_curve.control_stddev','lessthaneq',3,''));
doc_ids = db.search(query)
doc_ids = db.search(query, 'b5b')

[stevevanhooser]

I’ll get some time to look into this tomorrow morning, thanks

---

Stephen D. Van Hooser, PhD Associate Professor of Biology Brandeis University (+1) 781-736-2721 {www http://www.vhlab.org/,wiki http://wiki.vhlab.org/,code http://code.vhlab.org/,dataclass http://dataclass.vhlab.org/,twitter http://twitter.vhlab.org/}.vhlab.org http://vhlab.org/

On Aug 26, 2022, at 11:59 AM, Yair Altman @.***> wrote:

I just uploaded a commit with numerous fixes to both did.database and did.implementations.sqlitedb - please test it carefully, starting with the public methods of did.database. Here's a sample session:

db = did.implementations.sqlitedb('test2.sqlite'); db.add_branch('b5') db.add_branch('b5a') db.add_branch('b5b','b5') db.add_branch('b5b2','b5b') db.display_branches('b5') branch_ids = db.get_sub_branches('b5') db.set_branch('b5') branch_id = db.get_branch branch_id = db.get_parent_branch('b5b') db.add_doc(doc) db.add_doc(doc,'b5') db.add_doc(doc,'b5a') db.add_doc(doc,'b5b') doc_ids = db.get_doc_ids() doc_ids = db.get_doc_ids('b5a') query = did.query('speedtuning_calc.depends_on.name','exact_string','stimulus_tuningcurve_ID','').or(did.query('speed_tuning.tuning_curve.control_stddev','lessthaneq',3,'')); doc_ids = db.search(query) doc_ids = db.search(query, 'b5b') — Reply to this email directly, view it on GitHub https://github.com/VH-Lab/DID-matlab/issues/27#issuecomment-1228669835, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOOPDPX7R2TJ7S5J4RUHQLV3DSWDANCNFSM5SQJ5JMA. You are receiving this because you authored the thread.

[stevevanhooser]

Hi Yair -

I wrote some fancy test code and then I found that the code fails for me at the first stage.

When I write

db = did.implementations.sqlitedb('test2.sqlite'); db.add_branch('b5') I get an error saying that the branch name must not be empty. Not sure what’s going on. I also started a test file called did.test.test_did_branches It makes a random tree (see did.test.fun.*) and then tries to add the tree to the database. One could then attempt to delete a branch and test that it matches the randomly generated tree. Want to try to finish off that test function? Thanks Steve

On Aug 26, 2022, at 11:59 AM, Yair Altman @.***> wrote:

I just uploaded a commit with numerous fixes to both did.database and did.implementations.sqlitedb - please test it carefully, starting with the public methods of did.database. Here's a sample session:

db = did.implementations.sqlitedb('test2.sqlite'); db.add_branch('b5') db.add_branch('b5a') db.add_branch('b5b','b5') db.add_branch('b5b2','b5b') db.display_branches('b5') branch_ids = db.get_sub_branches('b5') db.set_branch('b5') branch_id = db.get_branch branch_id = db.get_parent_branch('b5b') db.add_doc(doc) db.add_doc(doc,'b5') db.add_doc(doc,'b5a') db.add_doc(doc,'b5b') doc_ids = db.get_doc_ids() doc_ids = db.get_doc_ids('b5a') query = did.query('speedtuning_calc.depends_on.name','exact_string','stimulus_tuningcurve_ID','').or(did.query('speed_tuning.tuning_curve.control_stddev','lessthaneq',3,'')); doc_ids = db.search(query) doc_ids = db.search(query, 'b5b') — Reply to this email directly, view it on GitHub https://github.com/VH-Lab/DID-matlab/issues/27#issuecomment-1228669835, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOOPDPX7R2TJ7S5J4RUHQLV3DSWDANCNFSM5SQJ5JMA. You are receiving this because you authored the thread.

[altmany]

I just committed several fixes, please retest

[stevevanhooser]

The re-test passes well! I started a test function did.test.test_did_branches. I didn’t test all of the methods. I think we will eventually need to test its ability to identify the frozen branches, the branches that are or aren’t editable, etc.

Also, it would be really useful to write a test function that generates a bunch of did.documents with some dependencies. Perhaps this is something that I should do? Then you can integrate the documents into the did.test.test_did_branches function to make sure the document relationships are maintained across the branches?

On Sep 8, 2022, at 7:50 AM, Yair Altman @.***> wrote:

I just committed several fixes, please retest

— Reply to this email directly, view it on GitHub https://github.com/VH-Lab/DID-matlab/issues/27#issuecomment-1240610763, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOOPDIO6QDND4DTGRYIAITV5HHHLANCNFSM5SQJ5JMA. You are receiving this because you authored the thread.

[altmany]

I think that at this time it would be best that you create the tests, to ensure that the implementation that I created matches the expected usage flows, with which I am not familiar.

Also, it is always better that someone different than the developer creates the test suite, because developers typically develop the tests to match the code they have created, whereas an independent person might think of use-cases or usage patterns that the developer hasn't considered.

I'm not going away and I will still be available for follow-ups as needed.

[stevevanhooser]

Hi Yair -

I finally got some time to make some test code.

Please run

[b,msg] = did.test.test_did_db_documents()

For me, it fails on adding the second document in the array of documents that were produced.

Also, I'm on the fence as to whether we should start the database with a top-level branch 'a'. It seems I need to call set_branch right after add_branch (add_branch by itself doesn't set the branch but it feels like maybe it should if one is making the first and only branch).

What do you think?

And you can check out the did.test.test_did_db_documents and see why it isn't adding more than one document in the set that is randomly generated.

(The next thing I was going to do was to add a branch onto 'a' called 'a_a' and make sure the set of documents are found there, too.)

Thanks Steve

[altmany]

I just committed the fixes (4e2e437df9821f6566c307d7d0171f82fd15b98c) - the test function now completes successfully.

There were several types of problems:

1. There was an unnecessary call to db.set_branch('a') after the call to db.add_branch('a') in test.db.test_did_db_documents(). This is not actually an error, it is just unnecessary because unlike what you said in your comment, add_branch does indeed set the current branch after adding the new branch.
2. sqlitedb.m included incorrect restrictions on the uniqueness of certain repeating values, so when you added the 2nd doc with some duplicate fields it wouldn't let you.
3. The did.test.documents.verify_db_document_structure() function was incorrectly placed within a file called _verify_document_nodestructure.m - I renamed the file to have the same name as the function
4. The did.test.documents.verify_db_document_structure() function called db.get_doc() instead of db.get_docs(). Note that db.get_docs() accepts an array of IDs, not just a single ID.
5. The did.test.documents.verify_db_document_structure() function didn't initialize the internal flag b so the code croaked when it tried to access it in the loop.

[stevevanhooser]

Wonderful!

I’ve added a new test function test_did_db_documents_branch.m that is failing. It seems to have trouble deleting the documents from a new branch that is based on another branch. Can you check it out when you have a chance?

Thanks Steve

On Oct 9, 2022, at 10:44 AM, Yair Altman @.***> wrote:

I just committed the fixes (4e2e437 https://github.com/VH-Lab/DID-matlab/commit/4e2e437df9821f6566c307d7d0171f82fd15b98c) - the test function now completes successfully.

There were several types of problems:

There was a duplicate (unnecessary) call to db.set_branch('a') after the call to db.add_branch('a') in test.db.test_did_db_documents(). This is not actually an error, it is just unnecessary. sqlitedb.m included incorrect restrictions on the uniqueness of certain repeating values, so when you added the 2nd doc with some duplicate fields it wouldn't let you. The did.test.documents.verify_db_document_structure() function was incorrectly placed within a file called verify_document_node_structure.m - I renamed the file to have the same name as the function The did.test.documents.verify_db_document_structure() function called db.get_doc() instead of db.get_docs(). Note that db.get_docs() accepts an array of IDs, not just a single ID. The did.test.documents.verify_db_document_structure() function didn't initialize the internal flag b so the code croaked when it tried to access it in the loop. — Reply to this email directly, view it on GitHub https://github.com/VH-Lab/DID-matlab/issues/27#issuecomment-1272557458, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOOPDIPMYNBI7GGVCNEEDDWCLK6HANCNFSM5SQJ5JMA. You are receiving this because you authored the thread.

[altmany]

Fixed (a804accb923a048b89df35e23ee05e9218b68f74) - did.test.test_did_db_documents_branch() now completes successfully