Out of memory problem

[SMRFoundation]

Every time I try to get the followers of a Twitter account that have more than aprox. 150000 followers, NodeXL crash. My computer is a Core i3 with 4 gigas of RAM, Windows 7 and Office 2007. Here is the error details.   How can I solve this problem?   Thanks   Alejandro   Microsoft NodeXL An unexpected problem occurred. If it occurs again, please copy the details to the clipboard by typing Ctrl-C, then post the details to http://www.codeplex.com/NodeXL/Thread/List.aspx.       Details:       [OutOfMemoryException]: Exception of type 'System.OutOfMemoryException' was thrown.       at System.String.GetStringForStringBuilder(String value, Int32 startIndex, Int32 length, Int32 capacity)   at System.Text.StringBuilder.GetNewString(String currentString, Int32 requiredLength)   at System.Text.StringBuilder.Append(Char value)   at System.IO.StringWriter.Write(Char value)   at System.Xml.XmlTextWriter.WriteEndAttributeQuote()   at System.Xml.XmlTextWriter.AutoComplete(Token token)   at System.Xml.XmlTextWriter.WriteEndAttribute()   at System.Xml.XmlAttribute.WriteTo(XmlWriter w)   at System.Xml.XmlElement.WriteTo(XmlWriter w)   at System.Xml.XmlElement.WriteContentTo(XmlWriter w)   at System.Xml.XmlElement.WriteTo(XmlWriter w)   at System.Xml.XmlElement.WriteContentTo(XmlWriter w)   at System.Xml.XmlElement.WriteTo(XmlWriter w)   at System.Xml.XmlElement.WriteContentTo(XmlWriter w)   at System.Xml.XmlElement.WriteTo(XmlWriter w)   at System.Xml.XmlDocument.WriteContentTo(XmlWriter xw)   at System.Xml.XmlDocument.WriteTo(XmlWriter w)   at System.Xml.XmlNode.get_OuterXml()   at Microsoft.NodeXL.GraphDataProviders.GraphDataProviderBase.TryGetGraphData(String& graphDataAsGraphML)   at Microsoft.NodeXL.ExcelTemplate.ThisWorkbook.ImportFromGraphDataProvider(IGraphDataProvider graphDataProvider) OK

This work item was migrated from CodePlex

CodePlex work item ID: '20572' Vote count: '2'

[SMRFoundation]

[tcap479@4/25/2011] Alejandro:

NodeXL uses a chunk of memory for each follower it finds in the requested Twitter network. In your case, there are too many followers in the network and not enough memory in your computer to handle all of them.

You can try rebooting your computer, then running NodeXL with no other programs running. If that doesn't work, the only other solution is to increase the amount of memory in your computer, assuming it's a 64-bit version of Windows 7 and has the memory slots to support more than 4 GB.

But I should mention that NodeXL is not optimized for networks that large, and that even if you succeed in collecting your 150,000-vertex Twitter network, the program's performance is going to be poor. In particular, laying out and displaying the graph is going to be very slow. In general, I tell people that NodeXL works well with a few thousand vertices. You can push it with a few tens of thousands if you have some patience, but 150,000 is likely to be too slow for most uses.

-- Tony

[SMRFoundation]

[alegrego@4/25/2011] I´m using NodeXL to get the data from Twitter. For the analisys and the graph displaying I´m using Cytoscape and Gephi. How can I use NodeXL to only get the data from the Twitter API?

Thank you very much for the answer.

[SMRFoundation]

[tcap479@4/25/2011] I've created a new discussion for the Cytoscape question from alegrego. It's at http://nodexl.codeplex.com/discussions/255209.

-- Tony

[SMRFoundation]

[alegrego@4/26/2011] I just run NodeXL in a corei7, 8gb ram computer with 64-bit version of Windows 7 and also crashed. Perhaps the solution to the problem is that while downloading the data, NodeXL stores it locally in the excel file. This might reduce the amount of ram NODEXL uses while getting the data from Twitter.

Any other suggestions or ways to run the proccess in segments?

[SMRFoundation]

[tcap479@4/26/2011] Alejandro:

The Twitter part of the program could indeed be reengineered to scale more efficiently to large networks. It can't write directly to the Excel workbook, because the Twitter importers are actually "plug-in" components that have no direct connection to the workbook, but it could conceivably build the network in a file instead of in memory, thus avoiding the out-of-memory condition you're hitting.

However, even if we took the time to do that (and it's not a trivial task), NodeXL users would run into other bottlenecks further down the chain, such as when the graph is laid out and displayed. (I know that you're not using NodeXL for those tasks, but I think most other people are.) As I mentioned, NodeXL is not optimized for large networks, and many other parts of the program would have to be overhauled to allow it to handle large graphs. We opted for ease of use over massive scalability, and that emphasis is unlikely to change in the near future.

So I don't think you're going to be able get your 150,000-vertex Twitter network using NodeXL. But NodeXL is only one of many programs that use the Twitter API, so perhaps there is some other software out there that can do what you need to do.

-- Tony

[SMRFoundation]

[UnknownUser@6/24/2011]

[SMRFoundation]

[UnknownUser@2/21/2013]