search

crossminer / scava

https://eclipse.org/scava/
Eclipse Public License 2.0
18 stars 13 forks source link

NNTP reader should store the last article read #439

Closed creat89 closed 4 years ago

creat89 commented 4 years ago

Curently the reader tries to store the last article read from an NNTP in the MongoDB Collection. However, as far as I can determine, it is impossible to the reader to modify these entries. Thus, the reader can take a long time to read large NNTP repositories.

I commented the possible solution in:

https://github.com/crossminer/scava/commit/e0c52586ce84efe890280b68307c3b672b4216d7

creat89 commented 4 years ago

This has been solved in https://github.com/crossminer/scava/commit/e7477e25ce2cddccbbf9305a0817e5d725a2eeac

creat89 commented 4 years ago

This should improve the speed of the NNTP reader as well. Because in the original implementation, it was reading everything all the possible articles. Thus, the older the NNTP the longer it took to reach the expected date. The first time it might be slow, especially if between the start of the NNTP channel and the time of the delta is large. But after the first day, the NNTP should have a consistent speed.