add check for ignore for recently edited file

[PRANJALRANA11]

fix for #2734

added check for ignore for recently edited file

[sestinj]

@PRANJALRANA11 Thanks for taking the time to do this! This is going to be the perfect band-aid for small codebases, but I'm trying to think ahead to what happens when there is a huge codebase: with this implementation, we would walk the entire directory every time the user changes files, which is very expensive.

I'm thinking we'll need to instead have a way of storing the .gitignore and .continueignore locations, or instead just walking up the file tree from a given location to find them. This is a bit more involved, but in the meantime what you could do is use the default ignore patterns in ignore.ts (which include .env, etc.) here and it would solve this problem for 90% of users

[RomneyDa]

@sestinj good point, would be bad for large repos @PRANJALRANA11 can you check ignore for just the default ones and I'll open a separate branch for the walking?

https://github.com/continuedev/continue/blob/dev/core/indexing/walkDir.ts

ignore().add(defaultIgnoreFile)
ignore.ignores(path)

Future will need to think about using "didChangeActiveTextEditor" for this, that could be triggered 20 times a second, can't do anything expensive here

[PRANJALRANA11]

@PRANJALRANA11 Thanks for taking the time to do this! This is going to be the perfect band-aid for small codebases, but I'm trying to think ahead to what happens when there is a huge codebase: with this implementation, we would walk the entire directory every time the user changes files, which is very expensive.

I'm thinking we'll need to instead have a way of storing the .gitignore and .continueignore locations, or instead just walking up the file tree from a given location to find them. This is a bit more involved, but in the meantime what you could do is use the default ignore patterns in ignore.ts (which include .env, etc.) here and it would solve this problem for 90% of users

If we try to do some sort of memorization here it then ! I think the no. Of times walkdir run will greatly reduced, it will only run one time for a codebase and then we can just loop around the results

[PRANJALRANA11]

@sestinj @RomneyDa I have tried some sort of memoization func and now the walk dir only runs when workspace path changes else it gives out the memoized results lmk if that works

[RomneyDa]

Maybe add a TODO comment about adding an ignored files cache that is cleared/filtered on ignore file updates, about speed concerns, etc.

[PRANJALRANA11]

ok sure

[RomneyDa]

@PRANJALRANA11 apologies for the stop n go on this one. Bumping priority on this one, do you think you'll have a chance to knock out just the defaultIgnoreFile ignores? If not I can snag in a few hours. Thanks!

[PRANJALRANA11]

@RomneyDa hey sorry man I'm in IST timezone so it was late night yesterday

[PRANJALRANA11]

Will resolve it today no worries

[RomneyDa]

Got it, no worries thanks!

[RomneyDa]

@sestinj tested, works. Benchmarked the entire event from start to cached at 4 ms, should be fine

[RomneyDa]

@PRANJALRANA11 Thank you!!