search

returneksibir / yakala

a humble web crawler framework
3 stars 1 forks source link

reason of slowness after long run and OutOfMemoryError #19

Open sardok opened 13 years ago

sardok commented 13 years ago

is Set(). the debug output shows that it is start to run giving errors around 1000000th (a million) entry. But the worse thing is, it is getting slower.

rimbi commented 13 years ago

What a nice news! Should we use another data structure?

sardok commented 13 years ago

imho we should avoid saving link in memory, im currently working on a database on a file. i'll commit to this actor-design if it successes. right now because of this issue i cannot do fully test on actor-design branch.

On Fri, Aug 19, 2011 at 4:14 PM, rimbi reply@reply.github.com wrote:

What a nice news! Should we use another data structure?

Reply to this email directly or view it on GitHub: https://github.com/returneksibir/yakala/issues/19#issuecomment-1850835

rimbi commented 13 years ago

Let's fix it on another branch?

On Fri, Aug 19, 2011 at 4:25 PM, sardok < reply@reply.github.com>wrote:

imho we should avoid saving link in memory, im currently working on a database on a file. i'll commit to this actor-design if it successes. right now because of this issue i cannot do fully test on actor-design branch.

On Fri, Aug 19, 2011 at 4:14 PM, rimbi reply@reply.github.com wrote:

What a nice news! Should we use another data structure?

Reply to this email directly or view it on GitHub: https://github.com/returneksibir/yakala/issues/19#issuecomment-1850835

Reply to this email directly or view it on GitHub: https://github.com/returneksibir/yakala/issues/19#issuecomment-1850899

sardok commented 13 years ago

ok.

On Fri, Aug 19, 2011 at 5:09 PM, rimbi reply@reply.github.com wrote:

Let's fix it on another branch?

On Fri, Aug 19, 2011 at 4:25 PM, sardok < reply@reply.github.com>wrote:

imho we should avoid saving link in memory, im currently working on a database on a file. i'll commit to this actor-design if it successes. right now because of this issue i cannot do fully test on actor-design branch.

On Fri, Aug 19, 2011 at 4:14 PM, rimbi reply@reply.github.com wrote:

What a nice news! Should we use another data structure?

Reply to this email directly or view it on GitHub: https://github.com/returneksibir/yakala/issues/19#issuecomment-1850835

Reply to this email directly or view it on GitHub: https://github.com/returneksibir/yakala/issues/19#issuecomment-1850899

Reply to this email directly or view it on GitHub: https://github.com/returneksibir/yakala/issues/19#issuecomment-1851223