code4craft / webmagic

A scalable web crawler framework for Java.
http://webmagic.io/
Apache License 2.0
11.37k stars 4.18k forks source link

Refactored Code to Resolve Implementation Code Smells #1151

Closed ayushi250317 closed 5 months ago

ayushi250317 commented 5 months ago
  1. Cyclomatic complexity of method processSingle present in PageModelExtractor class was 19. Reduced it to 9 by introducing two methods that handles extraction of single value and multi values from source.
  2. Renamed convert methods in PageModelExtractor class for increasing code readability
  3. In convertHttpClientContext method of HttpUriRequestConverter class, introducing explaining variable to simplify the complex statement : authState.update(new BasicScheme(ChallengeState.PROXY),new UsernamePasswordCredentials(proxy.getUsername()proxy.getPassword())) and increasing code readability