extendio was the first largish project I wrote in go, so I made many mistakes; although in its current state it does what it was designed to do, but the implementation needs a re-write incorpating the lessons learnt from experience. One of the biggest problems in that nav is too big. There are many features all mixed together. We should have more sub packages which are more focused/cohesive.
Actually, I think there is a good case to make a new repository walker (hike, stride, march, ramble, tread) that only does navigation, and leave extendio to be a platform package.
Things to improve
have a new "trav" top level package (to replace nav) which has many sub packages, eg filter, resume, depth, notify, hooks, async, persist(marshalling), measure (metrics), sample, options, listen, enum (actually enum should stay in extendio)
make sure that log and config are created as soon as possible within the bootstrap phase
no inheritance (the agent idea was a poor choice and just reflected my OO background, not making the full switch away from inheritance). We can still have an agent (but should be renamed to be a verb rather than a noun) but its just a function to which you pass a navigator, no inheritence is required
change the virtual file system to be compatble with os.fs
make sure all file system operations go via the virtual file system rather than using native directly
improve the bootstrap process (maybe dependency injection); define a set of clear phases and stick to them so it clears what happens and when; to get rid of the confusion about what type of operation can happen when
improve the filters with respect to sampling. The sampling feature was much more complicated than it needed to be because of filtering; see issue: #373 (FILTERING MUST HAPPEN BEFORE SAMPLING, this will eliminate a lot of the complexity, which manifested itself in the lookahead/caching mechanism that was desgned in as a work-around)
use code genertors for i18n errors and compose them in such a way so they can be identified by errors.Is
the LISTEN decorator must precede the FILTER decorator, that way, we can listen for an event that can't be filtered out, by accident, or perhaps make this explicit. It may be that LISTEN should only be dependent of events that are filtered in - this could be an option; actually impelement this in such a way that it is clear to see how this is compsed, ie in a functional/fluent way. Also consider this in the content of resume, which uses the listen feature.
improve the decoration process especially with respect to filtering, perhaps this could be improved via the observer
there is currently a silly bug with respect to filtering when a worker-pool is being used and the metrics. The file/folder counts can mis calculated, purely because of the way the decorators are composed. This should be an an easy bug to fix, but it isnt because of the current structure
ensure the context can be handled better without having to store it; perhaps define a WithContext operator/function, similar to the way its done in charm.log
navigation frame has some 'model' like properties, so introduce a model and clarify the purpose of the frame (actually, I have changed my mind about a model in the navigator, this is a client side concern only, but we should clarify the purpose of the frame as it is currently a bit muddy)
be VERY strict about errors, make consistent. If there is any chance an method/function can result in an error, perfer to return the error if it has external significance. I did not recognise the importance of this until fairly recently, and it's time to do something about it, in version 2
each filter should be able to have multiple patterns, combined logically with AND op
make use of internal packages more
get rid of the callback model for options, we only want 1 way of setting options and that is the provided options model
get rid of the refernce pointers, this was experiemental and of little value
get rid of the navigator/traverse confusion, use one form of the terminology and make it consistent, or make it clear the difference between navigator or traverser
build in cancellation feature from the start that includes for sequential processing
build in the observer concept to make filtering easier
come up with a better name for the listen feature (waypoint/hibernation) as that was mis-named; The key concept behind Listen is that client callback is invoked on a conditional basis, ie we don't start invoking until a particular condition is met; ie we find a specific fie or directory and we can also stop invoking when another condition is met.
make sure that listen feature is implemented to make resume easier; part of the rationale of the listen feature was for resume, but it I am not happy with the way these 2 feature interacted; should be improved
get rid of Enum/En from variable names (this is an anti-pattern)
get rid of item extension, there's no need for it
get rid of the nil concept, perhaps we borrow the option concept from rust, but rename it to optional
change options to work more like they are in charm glamour - TermRendererOption; NB: also consider reactive options style, perhaps we have both style mixed together; ultimately there will be a master options object that can be marshalled into json, for the purposes of resume. Filtering would be a reactive style options as it affects observable events, where as behaviours would be the functional style.
extendio was the first largish project I wrote in go, so I made many mistakes; although in its current state it does what it was designed to do, but the implementation needs a re-write incorpating the lessons learnt from experience. One of the biggest problems in that nav is too big. There are many features all mixed together. We should have more sub packages which are more focused/cohesive.
Actually, I think there is a good case to make a new repository walker (hike, stride, march, ramble, tread) that only does navigation, and leave extendio to be a platform package.
Things to improve
Things that are good and should be retianed:
Also see 🚀 awesome-go