search

thaljef / Pinto

Curate your own repository of Perl modules
https://metacpan.org/module/Pinto::Manual
66 stars 49 forks source link

Cannot allocate memory error (when pulling huge distros) #110

Closed melezhik closed 10 years ago

melezhik commented 11 years ago

When pulling huge distros, and Google::Ads::AdWords::Client is among them, pinto stuck into memory issues:

pinto@pinto:~$ PINTO_EDITOR=less pinto pull -v Google::Ads::AdWords::Client Pulling target Google::Ads::AdWords::Client~0 to stack bampo Descending into prerequisites for DTORRES/GOOGLE-ADWORDS-PERL-CLIENT-2.11.0.tar.gz Error during edit (less): exit value(0), signal(0), core_dump(0): Cannot allocate memory at /home/pinto/opt/local/pinto/lib/perl5/Term/EditorEdit.pm line 37. *\ There was an error editing /tmp/6_FZLRYseN Do you want to (c)ontinue, (a)bort, or (s)ave?

top:

top - 12:19:32 up 1 day, 20:15, 2 users, load average: 0.05, 0.06, 0.01 Tasks: 77 total, 1 running, 76 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.3%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 4061608k total, 4016340k used, 45268k free, 117132k buffers Swap: 0k total, 0k used, 0k free, 388736k cached

24671 pinto 20 0 3430m 3.2g 4756 S 0.0 83.2 0:27.24 /usr/bin/perl /home/pinto/opt/local/pinto/bin/pinto pull -v Google::Ads::AdWords::Client

I wonder in which stage allocation memory problem happens? Looking at commit message and I see that all pending data are here:

melezhik@pinto:~$ du -sh /tmp/6_FZLRYseN 388K /tmp/6_FZLRYseN

melezhik@pinto:~$ sudo head -n 40 /tmp/6_FZLRYseN

Pull DTORRES/GOOGLE-ADWORDS-PERL-CLIENT-2.11.0.tar.gz

#-------------------------------------------------------------------------------
# Please edit or amend the message above as you see fit.  The first line of the 
# message will be used as the title.  Any line that starts with a "#" will be 
# ignored.  To abort the commit, delete the entire message above, save the file, 
# and close the editor. 
#
# Changes to be committed:
#
# +[rf-] Class::Std                                      0.011 DCONWAY/Class-Std-0.011.tar.gz
# +[rf-] Class::Std::SCR                                     0 DCONWAY/Class-Std-0.011.tar.gz
# +[rf-] Class::Std::Fast                               v0.0.8 ACID/Class-Std-Fast-v0.0.8.tar.gz
# +[rf-] Class::Std::Fast::Storable                     v0.0.8 ACID/Class-Std-Fast-v0.0.8.tar.gz
# +[rf-] Config::Properties                               1.75 SALVA/Config-Properties-1.75.tar.gz
# +[rf-] Crypt::OpenSSL::RSA                              0.28 PERLER/Crypt-OpenSSL-RSA-0.28.tar.gz
# +[rf-] Crypt::OpenSSL::Random                           0.04 IROBERTS/Crypt-OpenSSL-Random-0.04.tar.gz
# +[rf-] Data::Uniqid                                     0.12 MWX/Data-Uniqid-0.12.tar.gz
# +[rf-] File::HomeDir                                    1.00 ADAMK/File-HomeDir-1.00.tar.gz
# +[rf-] File::HomeDir::Darwin                            1.00 ADAMK/File-HomeDir-1.00.tar.gz
# +[rf-] File::HomeDir::Darwin::Carbon                    1.00 ADAMK/File-HomeDir-1.00.tar.gz
# +[rf-] File::HomeDir::Darwin::Cocoa                     1.00 ADAMK/File-HomeDir-1.00.tar.gz
# +[rf-] File::HomeDir::Driver                            1.00 ADAMK/File-HomeDir-1.00.tar.gz
# +[rf-] File::HomeDir::FreeDesktop                       1.00 ADAMK/File-HomeDir-1.00.tar.gz
# +[rf-] File::HomeDir::MacOS9                            1.00 ADAMK/File-HomeDir-1.00.tar.gz
# +[rf-] File::HomeDir::TIE                                  0 ADAMK/File-HomeDir-1.00.tar.gz
# +[rf-] File::HomeDir::Test                              1.00 ADAMK/File-HomeDir-1.00.tar.gz
# +[rf-] File::HomeDir::Unix                              1.00 ADAMK/File-HomeDir-1.00.tar.gz
# +[rf-] File::HomeDir::Windows                           1.00 ADAMK/File-HomeDir-1.00.tar.gz
# +[rf-] Google::Ads::AdWords::AuthTokenHandler        v2.11.0 DTORRES/GOOGLE-ADWORDS-PERL-CLIENT-2.11.0.tar.gz
# +[rf-] Google::Ads::AdWords::Client                  v2.11.0 DTORRES/GOOGLE-ADWORDS-PERL-CLIENT-2.11.0.tar.gz
# +[rf-] Google::Ads::AdWords::Constants               v2.11.0 DTORRES/GOOGLE-ADWORDS-PERL-CLIENT-2.11.0.tar.gz
# +[rf-] Google::Ads::AdWords::Deserializer            v2.11.0 DTORRES/GOOGLE-ADWORDS-PERL-CLIENT-2.11.0.tar.gz
# +[rf-] Google::Ads::AdWords::FaultDetail             v2.11.0 DTORRES/GOOGLE-ADWORDS-PERL-CLIENT-2.11.0.tar.gz
# +[rf-] Google::Ads::AdWords::Logging                 v2.11.0 DTORRES/GOOGLE-ADWORDS-PERL-CLIENT-2.11.0.tar.gz
# +[rf-] Google::Ads::AdWords::OAuth2ApplicationsHandler      v2.11.0 DTORRES/GOOGLE-ADWORDS-PERL-CLIENT-2.11.0.tar.gz
# +[rf-] Google::Ads::AdWords::OAuth2ServiceAccountsHandler      v2.11.0 DTORRES/GOOGLE-ADWORDS-PERL-CLIENT-2.11.0.tar.gz
# +[rf-] Google::Ads::AdWords::Reports::DateRange            0 DTORRES/GOOGLE-ADWORDS-PERL-CLIENT-2.11.0.tar.gz
# +[rf-] Google::Ads::AdWords::Reports::DateRangeType            0 DTORRES/GOOGLE-ADWORDS-PERL-CLIENT-2.11.0.tar.gz

melezhik@pinto:~$ sudo tail -f /tmp/6_FZLRYseN

+[rf-] XML::XPath::Node::TextImpl 0 MSERGEANT/XML-XPath-1.13.tar.gz

# +[rf-] XML::XPath::NodeImpl                                0 MSERGEANT/XML-XPath-1.13.tar.gz
# +[rf-] XML::XPath::NodeSet                                 0 MSERGEANT/XML-XPath-1.13.tar.gz
# +[rf-] XML::XPath::Number                                  0 MSERGEANT/XML-XPath-1.13.tar.gz
# +[rf-] XML::XPath::Parser                                  0 MSERGEANT/XML-XPath-1.13.tar.gz
# +[rf-] XML::XPath::PerlSAX                                 0 MSERGEANT/XML-XPath-1.13.tar.gz
# +[rf-] XML::XPath::Root                                    0 MSERGEANT/XML-XPath-1.13.tar.gz
# +[rf-] XML::XPath::Step                                    0 MSERGEANT/XML-XPath-1.13.tar.gz
# +[rf-] XML::XPath::Variable                                0 MSERGEANT/XML-XPath-1.13.tar.gz
# +[rf-] XML::XPath::XMLParser                               0 MSERGEANT/XML-XPath-1.13.tar.gz

root@pinto:/home/melezhik# su - pinto pinto@pinto:~$ pinto --version /home/pinto/opt/local/pinto/bin/pinto version 0.087_02 (Getopt::Long::GetOptions version 2.38; Perl version 5.10.1)

thaljef commented 11 years ago

I think @akarelas or @hesco reported something similar. The memory runs out when Pinto launches your EDITOR to edit the commit message. So you can work around this by using the -m option to specify your commit message at the command line. Or just use -M to accept the default commit message.

I'll have to look at the code more, but perl may be forking when it launches the EDITOR. That could be doubling the amount of memory consumed. I don't know if that's how fork actually works, but it sounds like a reasonable explanation.

I assume you're running this inside a virtual machine (like Vagrant). Is it possible for you to give the VM a little more memory? I think 1024MB should be enough.

melezhik commented 10 years ago

Hi Jeff. Yes, -m is workaround for this and I use it now. I guess this happens when adding / pulling huge archives like Google::Ads::AdWords::Client ... It's not that critical, but is still pesky ...

thaljef commented 10 years ago

It looks like I fixed this (so you don't have to use -m) but it hasn't been released yet. I will make a release tomorrow.

melezhik commented 10 years ago

Great!

2013/9/29 Jeffrey Ryan Thalhammer notifications@github.com

It looks like I fixed this (so you don't have to use -m) but it hasn't been released yet. I will make a release tomorrow.

— Reply to this email directly or view it on GitHubhttps://github.com/thaljef/Pinto/issues/110#issuecomment-25315186 .

thaljef commented 10 years ago

I was wrong. I had confused this bug with #109. It is not yet fixed. Reopening...

thaljef commented 10 years ago

If you're using Pinto 0.098 or later and http://cpan.stratopan.com is your first upstream source, then Pinto will use the Stratopan locator service to find packages, rather than reading the entire 02packages file. This reduces the memory consumption considerably, which should fix this problem.

Unless you have changed the upstream source in your config.ini file, Pinto should start defaulting to http://cpan.stratopan.com as soon as you upgrade to 0.098 or later. If, for some reason, the Stratopan locator service fails, Pinto will fall back to reading the 02packages. And in that case, it will consume a whole bunch of memory, just like before.