search

jim256 / ebay

0 stars 0 forks source link

Update prior_run_timestamp only if spider runs successfully #10

Closed jim256 closed 4 years ago

jim256 commented 4 years ago

I'm not sure of the best way to implement this. Maybe it would make the most sense to update it after the first page of results is retrieved?

james-carpenter commented 4 years ago

Right now it waits until the end of the run as the spider is closing.

Would you want it to only update if there were 0 errors during processing?

jim256 commented 4 years ago

@james-carpenter Oh. I see. I guess so? I'm just trying to think of how to avoid the issue where it hits the namespace problem, or potentially the authentication problem. I'd hate for the next run(s) to miss the results in between if it was unable to run.

james-carpenter commented 4 years ago

Agreed. Worst case there is that it gets some duplicates from a prior run and does nothing on the update. That's a simple fix.