search

cucapra / pollen

generating hardware accelerators for pangenomic graph queries
MIT License
24 stars 1 forks source link

Better testing, more graphs #50

Closed anshumanmohan closed 1 year ago

anshumanmohan commented 1 year ago

This PR improves our testing infrastructure a tad. We are (still) doing differential testing against odgi, so agreeing with them is always a good sign, but the problem was that the existing graphs were, in many cases, not proving to be very interesting targets for the algorithms. For instance, crush was having no effect on any graph, flip was only flipping one path in one graph, etc.

I had hoped to generate a bunch of new graphs that would run our algorithms through their paces, but it turned out there wasn't a ton to do. Here's a summary of how the various algorithms are tested after this PR:

  1. chop takes a command-line argument n for the chopping-size to aim for. I test it by generating n randomly using a little snippet of Python: n is chosen from (2, the length of the graph's longest segment).
  2. crush_n converts "runs" of N into a single character N. To test it I have added three new files to test/handmade/.
  3. I have added three new files to test flip. We already disagreed with the odgi output for the graph note5, and we disagree with them on all the graphs that I have crafted. I will explore this further.
  4. I already had a reasonably nice way of testing validate: I take a gfa, randomly nix 90% of its Links, and then run validate on the newly-crafted gfa.
  5. The naive version of overlap, where the candidate paths are tested along their entire length, is tested exhaustively at present. The more interesting case, where we query a path along a certain fragment of its length, is not yet implemented in slow-odgi. See https://github.com/cucapra/pollen/issues/32#issuecomment-1499839056 for more.
  6. IMO degree, depth,flatten, and matrix are tested reasonably well by the existing graphs.
anshumanmohan commented 1 year ago

I'll point out a few things that I don't love and would love guidance on.

  1. Where to store the handmade files: I currently have them in test/handmade/, and this means that make clean does not delete them. If there's a cleverer/neater way, please let me know!
  2. How to run certain algorithms against certain files and still retain sanity: I currently do, for example, -turnt --save --env crush_oracle test/handmade/crush*.gfa and, under test/handmade/, I have crush1.gfa, crush2.gfa, and crush3.gfa. Similar with flip. The idea is not to run, say, flip, against the graphs that were made especially for crush.
anshumanmohan commented 1 year ago

There was one thing that I was seeking guidance on, but I've moved that into #46 and closed this PR.

susan-garry commented 1 year ago
  1. Where to store the handmade files: I currently have them in test/handmade/, and this means that make clean does not delete them. If there's a cleverer/neater way, please let me know!

I can't quite figure out which files you think make clean should remove; can't you just add rm commands to the clean target?

  1. How to run certain algorithms against certain files and still retain sanity: I currently do, for example, -turnt --save --env crush_oracle test/handmade/crush*.gfa and, under test/handmade/, I have crush1.gfa, crush2.gfa, and crush3.gfa. Similar with flip. The idea is not to run, say, flip, against the graphs that were made especially for crush.

I don't think there's anything wrong with running each graph across each algorithm; it doesn't seem to take too much time, and it improves the testing coverage of all of our algorithms. However, if you want to be able to run specific tests on an algorithm for debugging purposes, you could give the .gfa files an extra extension, e.g. .crush.gfa files are made specifically for crush, and turnt --env crush_test test/handmade/*.crush.gfa would run tests on just these files.