Rewriting the whole thing

[ersoykadir]

We know that the codebase is messy. I will rewrite it, moving from high level requirements for this tool to low level requirements. I believe starting with the backend and assuming the tool functionality exists, I can conjure the necessary functionalities of the tool more clearly. Also I must decide on the flow of the tool.

Outline for the tool flow:

• Acquire repo and requirements info from the user.
• Fetch SDA data from repo and reqs from the SRS.
  • Problems here, before we created jsons to store these data. I feel like this creates an unnecessary step, find if possible to directly save them to graphdb
  • DONE, data taken from api and saved to graphdb directly
• Create nodes on graph db for each SDA and req
• Two options here:
  • Finish the flow, On antoher flow, expect user to input which artifacts to trace and then build traces
  • Build traces with pre-provided trace method and finish.
  • This step had problems before, the node data for tracing was acquired from the json files, even though we also had same data in graphdb
  • Another problem, a graph structure with python classes is built behind when tracing, check if this is necessary, is it possible to just get (text data, artifactnum) from db and provide traces? If not at least build python-class graph from db data.
  • DONE, we have two flows, one for creating nodes(along with natural(developer implied) links). Second is to let user decide artifact types and create traces. No extra python classes or jsons

For tracing:

• keep tf-idf and word-embed basics
  • DONE
• update word-embedding pretrained data to bert?(a better embedding info, cares contexts), or an embeddings data from an llm(just need to run once to get embeddigs)
  • DONE, needs improvements, finetuning with domain specific context or some other solutions
• For LLMs, we can get embeddings from them or directly use them for textual analysis(search)[https://safakkbilici.github.io/neural-information-retrieval/]
  • Hugging face has many pretrained llms(codeLLM, LLama), we can decide on a base LLM by experiment
  • Heard there is a method of finetuning LLMs with somewhat trained data(not fully labeled) (We might use cmpe451 repos for training)
  • Waiting to be addressed
• There is a problem of threshold decision ! Is there a way to normalize these similarity numbers? Or a way to find a good threshold number?
  • Waiting to be addressed

[ersoykadir]

• Issue-Issue natural connection(there might be issue-pr connections as well, checking)
  • trackedIssues
• Issue-pr natural connection might be stored here
  • linkedBranches
• Req-issue, req-pr natural connection
  • TODO: check req numbers in issue bodies before any tracing

UPDATE: Both work great. Even if some of the groups are not effectively using these github links, extracting the existing ones is valuable.

• TODO: Update graphql queries in code accordingly(already tried them on postman) An idea: Knowing these links are true links(?), can we use them as a guide in training based trace methods?

[ersoykadir]

A basis implemented:

• We can create databases for each owner-gitrepo pair
• We can populate the database with issue-pr-reqs

[ersoykadir]

Trace link infrastructure implemented.

• create trace links endpoint
• abstraction for allowing different trace methods to be added later on, tracer class

I can move on with implementing one of the trace methods, probably start with tfidf, since it has an implementation already in old repo. Then word-embeddings, again have implementation in old repo(might need some refinement). Finally I will try llms for tracing, this will need some research. Before moving to llms, I would like to implement a basic frontend and finish dockerization. Send this version to @uskudarli Next:

• Implement tfidf
  • For corpus, should i use only given source and target artifact types' nodes or all nodes?
  • I am not utilizing labels for now, should I, if so how? my document=title+body+comments
  • I dont know how to decide on threshold value! Research!
• Implement natural link finder for issue-pr, issue-issue (req-issue is easy as well, but postponing for now)
• Implement word-embeddings with bert

[ersoykadir]

Unexpectedly good results for now !

[ersoykadir]

• Frontend implemented
  • small issue -> when redirecting to neodash, there is no sense of authorization. Neodash allows any database to be connected.
  • I must use user system of neo4j if I want to restrict databases to each user
  • Or, just let everyone see and edit every db. Handle this issue later on, since it is not too critical!
• Don't forget to implement natural connection finding
  • issue-issue, issue-pr, req-issue, req-pr

Moving on with

• Dockerizing frontend, combining both front and back on single compose file
• implementing natural trace finders
• implementing word embedding method with BERT

[ersoykadir]

• Frontend dockerized, added to docker compose
• word embedding with bert added, needs to be tested before pushing

Questions:

• How to handle threshold problem, there is no definite answer,
  • Get rid of thresholds, just order similarity?
  • visual side gets out of hand, it needs to be thought thoroughly
  • Find a way to decide on a threshold?
  • User driven? Give couple of lists to user and let them pick last meaningful result, then average?
  • We have true links? Maybe use the average of their similarity? or make a classification task, and use them as labeled data?

NEXT:

• test word embeddings
  • After having a working flow of embeddings, test
  • different embeddings. We talked about getting embeddings from LLMs
    • maybe llms have better understanding of context? If not we can check if they can improve domain specifity(Software development domain)
  • Fine tuning embedding models for our task? Software development domain
• finding natural connections
• Reach to uskudarli, feedbacks needed

[ersoykadir]

• Word Embeddings flow tested, works
• Natural links issue-issue, issue-pr, req-issue, req-pr found
• threshold taken from user for both tfidf and embeddings

NEXT:

• test natural links and overall app flow
  • send the repo link to uskudarli so that she can playaround
• discuss threshold problem with uskudarli
• finetuning an embeddings model
• reach uskudarli for feedbacks overall
• checkout llms

[ersoykadir]

Clean up this years requirements and provide them as a list for population, extra option to uploading req file

[ersoykadir]

Yehuu(Natural trace links)

[ersoykadir]

Dashboard, allow selecting requirement by number

[ersoykadir]

• Getting back to Actor-Action-Object structure
• Maybe we can put a weight function that gives more emphasis on these words on word embeddings?
  • Detect these elements of a requirement (POS tagging, NER, NEL) Entity recognition
• Inspect distribution of similarities of a req to decide on a threshold?? Maybe a drastic change point?

[ersoykadir]

Closing this as most of the ReBuilding the tool is done. The issues still need to be addressed:

Tracing

• Tracing with LLMs
  • For LLMs, we can get embeddings from them or directly use them for textual analysis search
  • Hugging face has many pretrained llms(codeLLM, LLama), we can decide on a base LLM by experiment
  • Heard there is a method of finetuning LLMs with somewhat trained data(not fully labeled) (We might use cmpe451 repos for training)
  • Remembered from conference something like (LLMs decide better when provided with similarity numbers ordered?)
• Threshold problem
  • Get rid of thresholds, just order similarity?
  • visual side gets out of hand, it needs to be thought thoroughly
  • Find a way to decide on a threshold?
  • User driven? Give couple of lists to user and let them pick last meaningful result, then average?
  • We have true links? Maybe use the average of their similarity? or make a classification task, and use them as labeled data?
  • Inspect distribution of similarities of a req to decide on a threshold?? Maybe a drastic change point?
• Embeddings improvements
  • different embeddings. We talked about getting embeddings from LLMs
  • maybe llms have better understanding of context? If not we can check if they can improve domain specifity(Software development domain)
  • Fine tuning embedding models for our task? Software development domain
• Getting back to Actor-Action-Object structure
  • Entity extraction
  • POS - verb/noun
  • Entity Recognition
    • NER-NEL, Named entity Linking (Wikipedia-Wikidata)
    • TagMe-Sobigdata
      • spot, rho, link, p-link, hacing rho>0.3 and plink>0.1
    • Spacy-Nltk NEL
  • Maybe we can put a weight function that gives more emphasis on actions/objects on word embeddings?
  • Detect these elements of a requirement (POS tagging, NER, NEL) Entity recognition

    Dasboard/Tool

• Neo4j authorization
• Dashboard, allow selecting requirement by number