parse_dependencies removes filepaths

[leilani-reich]

Refer to https://github.com/sailuh/kaiaulu/pull/171#issuecomment-1493889187.

While testing my dependencies_to_sdsmj() function, I checked the unique filepaths from the combined src and dest files from the parsed depends table, which I got from parse_dependencies(), and I compared these filepaths to the filepaths in the "variables" field in the sdsm.json directly from the Depends command. However, there are less unique filepaths from my parsed depends table than there were in the Depends sdsm.json, so it seems like the parse_dependencies() function is removing filepaths.

Here are the filepaths that my depends table is missing:

 [1] "helix-common/src/main/java/org/apache/helix/ZNRecordDelta.java"                              
 [2] "helix-core/src/main/java/org/apache/helix/controller/package-info.java"                      
 [3] "helix-core/src/main/java/org/apache/helix/controller/pipeline/package-info.java"             
 [4] "helix-core/src/main/java/org/apache/helix/controller/stages/package-info.java"               
 [5] "helix-core/src/main/java/org/apache/helix/examples/package-info.java"                        
 [6] "helix-core/src/main/java/org/apache/helix/manager/zk/package-info.java"                      
 [7] "helix-core/src/main/java/org/apache/helix/messaging/handling/package-info.java"              
 [8] "helix-core/src/main/java/org/apache/helix/messaging/package-info.java"                       
 [9] "helix-core/src/main/java/org/apache/helix/model/package-info.java"                           
[10] "helix-core/src/main/java/org/apache/helix/monitoring/mbeans/package-info.java"               
[11] "helix-core/src/main/java/org/apache/helix/monitoring/package-info.java"                      
[12] "helix-core/src/main/java/org/apache/helix/package-info.java"                                 
[13] "helix-core/src/main/java/org/apache/helix/participant/package-info.java"                     
[14] "helix-core/src/main/java/org/apache/helix/participant/statemachine/package-info.java"        
[15] "helix-core/src/main/java/org/apache/helix/spectator/package-info.java"                       
[16] "helix-core/src/main/java/org/apache/helix/store/PropertyChangeListener.java"                 
[17] "helix-core/src/main/java/org/apache/helix/store/package-info.java"                           
[18] "helix-core/src/main/java/org/apache/helix/store/zk/PropertyItem.java"                        
[19] "helix-core/src/main/java/org/apache/helix/store/zk/ZkListener.java"                          
[20] "helix-core/src/main/java/org/apache/helix/store/zk/package-info.java"                        
[21] "helix-core/src/main/java/org/apache/helix/tools/commandtools/JmxDumper.java"                 
[22] "helix-core/src/main/java/org/apache/helix/tools/package-info.java"                           
[23] "helix-core/src/main/java/org/apache/helix/util/package-info.java"                            
[24] "helix-core/src/test/java/org/apache/helix/TestGetProperty.java"                              
[25] "recipes/rsync-replicated-file-system/src/main/java/org/apache/helix/filestore/ChangeLog.java"
[26] "recipes/rsync-replicated-file-system/src/main/java/org/apache/helix/filestore/Test.java"     

I have attached a zip with the code I was using to check the difference between the sdsm.json and my dependencies_to_sdsmj() json, "helix.json" which contains the output of running the Depends command, and "checkResultApril3.json", which contains my dependencies_to_sdsmj() generated json.

parse_dependencies_removes_filepaths.zip

[carlosparadis]

For the record, was this addressed on 851097e by reading the variables field out of the dsm instead of incorrectly inferring it out of the edgelist?

https://github.com/sailuh/kaiaulu/blob/174c435cafaf2ecb1f6a7bab3cbab72f582120ae/R/parser.R#L598

Or is it still causing mismatch? I had a faint memory you told me the gitlog_to_hdsmj was the only one pending now, but I may be mixing.

[leilani-reich]

Hi yes, from what I've tested, the output of my depends_to_sdsmj() matches to DV8's in terms of number of variables, order & number of cells, and values of parameters.

Also, I mentioned before, but I again wanted to note that the order of the values parameters in my sdsm.json does not match DV8's, and that is because I noticed the parameters in the DV8 json do not have the same order all the time. For example, in the sdsm.json exported from Depends below, we can see that the "Call" and "Import" parameters appear in different orders for the two different cells.

Not to mention, I had emailed the DV8 team about this before and they confirmed this wasn't a big deal. Just wanted to be clear on that point.

Also, to make sure, it's alright that the sdsm.json I implemented uses shortened filepaths (since parse_dependencies shortens filepaths) as opposed to DV8, which uses the entire filepath?

For example, in DV8 this filepath is: /Users/xylent1/github/ics496/helix/helix-admin-webapp/src/main/java/org/apache/helix/webapp/HelixAdminWebApp.java

Whereas in my sdsm.json output it is: /helix-admin-webapp/src/main/java/org/apache/helix/webapp/HelixAdminWebApp.java

[carlosparadis]

Also, to make sure, it's alright that the sdsm.json I implemented uses shortened filepaths (since parse_dependencies shortens filepaths) as opposed to DV8, which uses the entire filepath?

Yes, I believe that is fine. Out of curiosity, when you tested DV8 with your milestone 3.4 functions, did the intermediate files generated by DV8 also contained short paths, or did somehow they end up with full paths?

[leilani-reich]

Can you clarify what you mean by the intermediate files generated by DV8?

If I use the DV8 command to convert my sdsm.json output from depends_to_sdsmj() back to a dv8-dsm and then use another DV8 command to export the dv8-dsm as a sdsm.json again, the filepaths are still shortened in that sdsm.json. I'm not sure that is what you were asking though....

[carlosparadis]

I guess the less obscure way for me to put this is: If I had used your functions at the start of the DV8 Notebook Rick went over today, would the Architectural Flaws table (and everything in between, aka intermediate files in the pipeline) also contain shortened path files?

What I am trying to get at is, I don't think DV8 cares for the path being full or not. It will take what you give to it, and eventually output them on all commands in the same manner. If that is the case, then great! As the rest of the files in Kaiaulu should also (if not yet) be all shortened paths, so we can inner join without hassle. Full paths are problematic when multiple collaborators are sharing data, since full paths will not match, but relative paths following Kaiaulu convention should.

[leilani-reich]

Just tested it on the DV8 showcase. The intermediate files also contain shortened path files.

[carlosparadis]

Then perfect :) Since the intent is for us always to use your functions as they provide us with means to use the filter functions.

Since you confirm no further problem with this function, I am closing the issue. Feel free to re-open if you detect any other concerns here!