search

strohne / Facepager

Facepager was made for fetching public available data from YouTube, Twitter and other websites on the basis of APIs and webscraping.
https://github.com/strohne/Facepager/releases
506 stars 198 forks source link

extremely slow to delete large number of nodes in bulk #58

Closed world33 closed 7 years ago

world33 commented 7 years ago

Hi,

When I try to delete around 7,000 nodes in bulk it takes up to 30 minutes. Any chance to improve or optimize the deletion process in future releases? Thanks for the consideration.

strohne commented 7 years ago

Well, the main focus is on getting data, not on deleting it ;)

Deleting is slow and I am not sure whether we will work on this in the nearer future. Until then I suggest deleting the whole database or if you really need it using some SQLite-Browser for deleting.

world33 commented 7 years ago

Hehe,

Ok no worries I will delete the entire database if there is no other simple solution. What about adding a link to the database and custom presets folder in the UI in the next version if possible :-) ? I currently have to search in windows explorer for the file location each time I have to edit, delete a preset (I know I could create a shortcut...) I love your software and consider these only suggestions for further improvement.

strohne commented 7 years ago

Great idea, I will try to integrate links to the folders. I am curious: what type of project/research are you working on with Facepager?

world33 commented 7 years ago

Hi,

I am going to add a list of top 200 universities in the world based on the number of fans of their official facebook pages. The list of University facebook pages is available at http://www.4icu.org/facebook/ Plus I will extract contact addresses, year of foundation etc. and compare it with what we already have in the uniRank database. Your tool is very useful and before I was using only scrapebox

elektrotiko commented 7 years ago

Dear Strohne. First of all, let me thank you wholeheartedly for creating such a great tool. I experience problems with selecting a few hundreds of nodes in my sequential steps to extract data from FB. The program does not respond and I am forced to shut it down everytime I need to EXPAND and select all nodes (for instance, at 3 levels). Is there any workaround to this problem?

Perhaps one way of solving such problems would be to add a menu button for selecting all nodes at a particular level at once?

strohne commented 7 years ago

Thank you for your warm thank you.

Maybe you can omit expanding all nodes: just select the top level nodes and increase the node level in the settings. Data fetching is processed only for childnodes with the selected level.

Removing columns in the column setup could speed up expanding nodes. Expanding, selecting and deleting nodes are some of the weak points of the app, sorry. We lack resources for optimizing this.

How many nodes do you have alltogether? In the end, when working with many nodes the workaround is patience ;) While the interface is frozen, all resources are used for processing the task in the background.

Am 22. August 2017 20:07:28 MESZ schrieb elektrotiko notifications@github.com:

Dear Strohne. First of all, let me thank you wholeheartedly for creating such a great tool. I experience problems with selecting a few hundreds of nodes in my sequential steps to extract data from FB. The program does not respond and I am forced to shut it down everytime I need to EXPAND and select all nodes (for instance, at 3 levels). Is there any workaround to this problem?

-- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/strohne/Facepager/issues/58#issuecomment-324106901

-- Diese Nachricht wurde von meinem Android-Gerät mit K-9 Mail gesendet.

elektrotiko commented 7 years ago

Dear Strohne.

Maybe you can omit expanding all nodes: just select the top level nodes and increase the node level in the settings. Data fetching is processed only for childnodes with the selected level

Your above advice solved my problem! Thank you so much!