getappmap / appmap-js

Client libraries for AppMap
49 stars 17 forks source link

Navie doesn't find existing code #1891

Open apotterri opened 4 months ago

apotterri commented 4 months ago

I asked Navie for help making a change to some existing code. It didn't find the existing code, so it made up an answer.

AppMap Services log: navie-log.txt

Navie Context: [navie-context.txt](https://github.com/user-attachments/files/16313660/navie-context.txt]

The code that it needed to modify is here: https://github.com/getappmap/appmap-python/blob/a6eee3e5bcffb96f41b153ac5e6cfb3f5082aca4/_appmap/unittest.py .

dustinbyrne commented 4 months ago

Here are the vector terms for these prompts. Our static analysis will index file names and the names of functions and classes. These vector terms will be used to search that index.

How does the code disable recording of pytest test functions? ```txt disable +recording pytest test functions ```
Yes, I'm specifically interested in 3. Disabiling Utility Function. Can you give me more details about what that does? ```txt disable recording pytest test function +disabling utility ```
Can you show me how to add _disable_test_case to the unittest runner, in the function _appmap.unittest.testPartExecutor? ```txt disable recording pytest test functions utility function unittest runner appmap testpartexecutor case ```
dustinbyrne commented 4 months ago

Given this information, what were we missing in order to identify the correct file? To my eyes, the last prompt should have hit an exact match on testpartexecutor.

kgilpin commented 4 months ago

Some suggestions:

My guess is that testPartExecutor is being indexed as [ test, part, executor ] which is not matching the input word testpartexecutor.

When the index logic sees a camelized word like testPartExecutor, it will split that into its component words. But if it sees testpartexecutor it's not able to do that.

We should ensure that function terms like testPartExecutor are indexed as [test, part, executor, testpartexecutor ], and also press the vector terms service to generate cased search terms like testPartExecutor so that the de-camelizer will be able to separate this into different words.

kgilpin commented 4 months ago

I've also been thinking that we may want to change the way that we search for AppMap related code snippets.

The symbol-based search index actually post-dates the AppMap-based search index. So, once we have identified relevant AppMaps, we could boost the functions that are referenced by those AppMaps and use the symbol-based search index, and remove the EventIndex.