search

elastic / enterprise-search-network-drive-connector

Official Enterprise Search | Workplace Search - Network Drives Connector
Other
6 stars 3 forks source link

implement indexing logic #7

Closed praveen-elastic closed 2 years ago

praveen-elastic commented 2 years ago

Things included as a part of the PR:

Indexing rules Indexer Full Sync Incremental Sync Multiprocessing

praveen-elastic commented 2 years ago

@tarekziade, @artem-shelkovnikov I have updated the multithreading approach based on our discussion i.e. adding a queue in between fetching from the Network Drives and indexing to the Enterprise Search. However, there's one outstanding issue in Windows that we are encountering : Since we are passing the class variables while initializing the Process, in the Windows OS, it throws Pickle error since it won't be able to pickle the class variables/objects. ref: https://github.com/elastic/enterprise-search-network-drives-connector/blob/praveen/indexing-functionality/ees_network_drive/incremental_sync_command.py#L51