Documentation part 3

[Alhajras]

10.08.2023

• [x] Serve statistics
• [x] Add the reason for exit: Timeout, error, max depth or max docs, max pages.
• [x] Unit tests
  • [x] qgram
  • [x] Indexing
  • [x] Some jasmin tests

    06.08.2023

• [x] Add more node
• [x] UNi ranking evaluation against parsehub

  05.08.2023

• [ ] Clean up:
  • [ ] Refine documentation
  • [x] Add advanced options to inspectors
  • [x] Add form validators to the inspector
  • [x] Add logo

30.07.2023

• [x] Using a tuple instead of a class increased the performance by 18%
  • [ ] 1 thread with document and no tuple took 3.3 minutes
  • [ ] How to solve the CAPTCHAs
• [ ] How to increase the performance of indexing
  • [] Movies DB
  • [ ] 4 threads: 2.7 min
  • [ ] 1 threads: 3.3 min

29.07.2023

• [x] More evaluation on test crawler
• [x] Fix duplication by using hashing
• [x] Read about distributed indexing source

23.07.2023

• [x] Pages open JS takes too long to respond I need to handle this.https://crawler-test.com/javascript/alert_box fixed UnexpectedAlertPresentException.
• [x] https://crawler-test.com/status_codes/status_100 requests.exceptions.ConnectionError
• [x] https://crawler-test.com/redirects/infinite_meta_redirect Needs fix
• [x] -https://crawler-test.com/redirects/two_step_redirect_loop_1 TooManyRedirects

  22.07.2023

  • [x] More evaluation
  • [x] Fix ROBOTS.txt file
  • [ ] Crawler test website robots.text exclude all links!!
  • [ ] I should fix running crawling locally
  • [ ] Fix actions to be applied on different crawlers

    16.07.2023

• [x] I can sort the result based on price
• [x] I want to crawl big products from duaglas
• [x] I want to enforce unique to disallow duplication in the DB

15.07.2023

• [x] I want to use regular expressions to clean up process like 50,55 $ so I can have a number to use it for boosting.
• [x] I have to think about an idea of how to clean up the texts.
  • [x] I want to use it as a list fo regulars.

14.07.2023

• [x] Refinements: Boost results with log or just add score.
• [x] Reorganize the fields in indexers and crawler dialog.

09.07.2023

• [x] Use LinkFragment
• [x] Refinements: Weight words.
• [x] Refinements: Boost results with log or just add score.
• [x] Reorganize the fields in indexers and crawler dialog.

08.07.2023

• [x] Use drop down for actions
• [x] Fix timeline size
• [x] Download CSV

02.07.2023

• [x] I am trying to crawl and index other sites than movies DB and fixing issues.
• [x] Make showing the browser configurable
• [x] Flaconi indexing when I search for 'for' the site crashes!
• [x] Indexers should be editable and deleted, also started and stopped and parallel.
• [x] Create new indexing does not work anymore!

01.07.2023

• [x] Added more states to indexing
• [x] I am trying to crawl and index other sites than movies DB and fixing issues.
• [x] Enable and diable actions
• [x] Including the images in the douglas I found that I need to use the actions to load images.

  26.06.2023

• [x] Using c ped: Running python setup.py install unbder the ped_c file
• [x] Natalie meeting.
• [x] Allow users to enter their dictionary name (All can be under the dictionaries directory)
• [x] Allow users to enter q-gram options use_synonym and q
• [x] Split the advanced options from the easy options
• [x] Add types to InspectorValue image, text, link
• [x] Support images on the results page
• [x] The indexing part is not cached. Only the suggestions are.
• [x] Do I want to remove the one/multi option? it makes no sense!
• [x] Convert docuemntId to the tuple and see if it is faster now.
• [x] Allow users to choose the indexing machine to run their crawler on (local or docker)
• [x] Indexing needs a progress status
• [x] Speed up indexing and fuzzy search using multithreading.
• [x] Refinements: Boost results if all words exist in a document.
• [x] Refinements: Boost results if ranking based on popularity is needed.
• [ ] The result page should be using pagination! Add a pagination option to the UI.
• Compare the cache in the thesis try the available_indexers api

25.06.2023

• Optimize server interaction by debouncing
• I want to use the PED in python, but it does not work!!

  24.06.2023

• [x] cache in Django is too slow. I should replace it; using a singletone seems like a good idea

  23.06.2023

• [x] Start working on Fuzzy search with q-grams
• [x] Support synynoms in UI

Deliver search-friendly JavaScript-powered websites video https://www.youtube.com/watch?v=PFwUbgvpdaQ&t=2131s&ab_channel=GoogleSearchCentral

19.06.2023

• [x] Add skip words list
• [x] Add skip words with a threshold like less than three letters
• [x] Render the result in the FE (I have to make a generic approach) TABLE

  18.06.2023

• I found that using the URL to fetch the result index needs to be corrected as one URL can have several documents, so I need to use the document id to retrieve the document elements like the image, description, etc.
• [x] Create a migration and test that I can use the documents.
• [x] Change indexing in a way that it depend on the document not the URL
• [x] Add options to toggle b and a parameters.
• [x] Ranking now is working as is should.
• [x] I still want to render the result in Angular
• [x] Boost result if all words exist in a document.
• [x] Indexing options how many results should be shown? 25
• [x] After using the DocuemntScore object, the indexing is becomesing too slow :(
• [x] I have to watch the lecture about improving performance

17.06.2023

• [x] Multiable selector
• [x] Use the scope as a pagination method
• [x] Create Movies migration
• [x] I might want to create a model called document, where inspector values should save the document id as a foreign key.
• [x] Add the docs to the new migrations
• [x] Think about designing indexing and ranking and fuzzy search configurations.