Problem Installing PriorityQueue

[mme89]

hi guys, having trouble getting SecGen running on latest Kali Linux. I know not directly related to SecGen but couldn't find anything related anywhere else.

Installing PriorityQueue 0.1.2 with native extensions
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /tmp/bundler20190323-22771-xmpfy0PriorityQueue-0.1.2/gems/PriorityQueue-0.1.2/ext/priority_queue
/usr/bin/ruby2.5 -r ./siteconf20190323-22771-arw9zq.rb extconf.rb
creating Makefile

current directory: /tmp/bundler20190323-22771-xmpfy0PriorityQueue-0.1.2/gems/PriorityQueue-0.1.2/ext/priority_queue
make "DESTDIR=" clean

current directory: /tmp/bundler20190323-22771-xmpfy0PriorityQueue-0.1.2/gems/PriorityQueue-0.1.2/ext/priority_queue
make "DESTDIR="
compiling priority_queue.c
In file included from /usr/include/ruby-2.5.0/ruby.h:33,
                 from priority_queue.c:24:
priority_queue.c: In function ‘pq_node2dot’:
/usr/include/ruby-2.5.0/ruby/ruby.h:1779:3: error: call to ‘rb_varargs_bad_length’ declared with attribute error:  argument length doesn't
match
   rb_varargs_bad_length(argc, vargc)), \
   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/ruby-2.5.0/ruby/ruby.h:2474:6: note: in expansion of macro ‘rb_varargs_argc_check’
      rb_varargs_argc_check(rb_funcall_argc, rb_funcall_nargs), \
      ^~~~~~~~~~~~~~~~~~~~~
priority_queue.c:820:4: note: in expansion of macro ‘rb_funcall’
    rb_funcall(Qnil, id_format, 4, rb_str_new2("NODE%i -> NODE%i;\n"),
    ^~~~~~~~~~
make: *** [Makefile:243: priority_queue.o] Error 1

make failed, exit code 2

Gem files will remain installed in /tmp/bundler20190323-22771-xmpfy0PriorityQueue-0.1.2/gems/PriorityQueue-0.1.2 for inspection.
Results logged to /tmp/bundler20190323-22771-xmpfy0PriorityQueue-0.1.2/extensions/x86_64-linux/2.5.0/PriorityQueue-0.1.2/gem_make.out

An error occurred while installing PriorityQueue (0.1.2), and Bundler cannot continue.
Make sure that `gem install PriorityQueue -v '0.1.2' --source 'https://rubygems.org/'` succeeds before bundling.

In Gemfile:
  huffman was resolved to 0.0.1, which depends on
    PriorityQueue

[PrincessChalet]

I'm having the same issues. Have you been able to fix it?

[yudhaprimadiansyah]

i have the same issue,

[cliffe]

I've not tried installing on Kali, but you could try simply removing the huffman line from the Gemfile, and except for a few scenarios/modules that use that library SecGen should otherwise work.

[yudhaprimadiansyah]

that worked like a charm, thanks

[ghost]

that worked like a charm, thanks

What kali version are you using for SecGen? Im having trouble with alot of gems when running bundle install and wonder where to find the needed installations for kali.

[yudhaprimadiansyah]

that worked like a charm, thanks

What kali version are you using for SecGen? Im having trouble with alot of gems when running bundle install and wonder where to find the needed installations for kali.

I use kali linux version 2019.1, and i use ruby version 2.6.3, and add # into 'huffman' packages line in GemFile

[miguescri]

Sorry for reopening the thread, but this I think this is a problem that prevents installation in general in current Debian based distributions.

I'm trying to install it in a Ruby Docker container based in Debian Buster and it gives the same error. The PriorityQueue library dates back to 2009, so maybe it is related to backward compatibility with newer versions of Ruby.

Should in this case Huffman gem be removed from master and its dependent modules/scenarios be updated? I can take a look at it if you give me some hints.

[0xdreadnaught]

The latest build still won't install on kali 2019 or 2020. You can make a good amount of progress by downgrading Ruby to 2.6.3 with gem update --system 2.6.3 but things like ruby-dev are stuck at 2.7+1 due to the repos. I'm curious as to what platform Cliffe is building this on so I can spin up a VM and get it working. I don't expect a fix from the Ruby devs any time soon.

With Ruby 2.6.3 and ruby-dev 2.7+1, you can get it to finish bundle install by commenting out digest-whirlpool, packetfu, and huffman. But at that point, hardly any of the pre-built scenarios build.

[firewallzer0]

@DreadnaughtSec I just set this up on Ubuntu 20.04 I did this by uninstalling ruby and then using Ruby Version Manager (RVM) https://rvm.io/ I setup a 2.6.5 environment and got bundle to complete the install minus the huffman module. It still fails.

[0xdreadnaught]

@firewallzer0 Yea I went the RVM route as well. Ruby 2.6.3 had the best results, but it was only able to build out the default scenario. Everything other scenario fails to build. I've given up on using SecGen until either the Author updates it to run on a current Ruby version, or the Ruby devs fix their latest updates.

[cliffe]

Thanks for sharing. FYI we are using Ubuntu 16.04 and everything is working, with minimal effort. At some stage it would be great to get dependencies sorted for other distros and versions.

[cliffe]

Reopening since its still an issue affecting various distros

[0xdreadnaught]

16.04 might work for an existing installation that has a thorough apt cache built up. But it you install a fresh 16.04 then check the apt policy for things like ruby-dev you can't get roll back far enough to get things working 100%. I would definitely put the blame on the Ruby devs as opposed to you. The more I look into it, there are numerous cases around certain functions not working in 2.7 and their devs basically went "ok wtvr". Several of those functions are required for SecGen to work properly.

Would it be possible to get a ova uploaded somewhere of a Ubuntu 16.04 with a base secgen clone?

[thompslu]

Hi DreadnaughtSec, I’ve had no problems With a fresh install of 16.04 over the past couple of weeks so long as you sudo apt- get update / sudo apt-get upgrade and ensure 16.04 is up to date in every possible way before installing the required packages for the SecGen to run. I don’t use the provided vagrant as this now has issues instead I use version 2.2.6. Once this has installed Run all the commands provided in the instructions. Ruby 2.5.7 October 2019 release is working for me. I would recommend running it in a VM to ensure you can revert back without damaging your bare metal machine. An OVA file with 16.04 in its configured dev state to the point where the system is Mature with appropriate old repos that are not available anymore or at least harder to get would be good. But when it comes to updating the dev machine again it would need a lot of maintenance. I’m currently studying at university and over the past month or so I’ve been looking at upgrading the gems in the bundle Install to support the 18.04 LTS. I’m finishing off some of end of semester modules in the next few weeks but after this I will certainly be actively reviewing code systematically to ensure if they’re any issues there here to shared with fellow SecGen users. Let me know how you get on.

[0xdreadnaught]

Thompslu,

So I took another crack at it. Here are the results: [edit: this is a fresh 16.04 install]

sudo apt-get update
sudo apt-get upgrade
sudo apt-get full-upgrade
reboot
wget https://releases.hashicorp.com/vagrant/2.2.6/vagrant_2.2.6_x86_64.deb
sudo apt-get install ./vagrant_2.2.6_x86_64.deb
sudo apt-get install ruby-dev zlib1g-dev liblzma-dev build-essential patch virtualbox ruby-bundler imagemagick libmagickwand-dev exiftool libpq-dev libcurl4-openssl-dev libxml2-dev graphviz graphviz-dev libpcap0.8-dev git
cd /opt
sudo git clone https://github.com/cliffe/SecGen
sudo mkdir -p /home/user/bin
sudo cp -r ./SecGen /home/user/bin/
cd /home/user/bin/SecGen
bundle install
sudo mkdir ./projects
sudo chmod 777 ./projects
ruby secgen.rb run
success

flawed_fortress_1 fails to build:

Running tests for unrealirc_3281_backdoor
FAILED: Port 6667 is closed at 172.16.0.5 (in_the_wild)!
Post provision tests contained failures!

Post provision tests contained failures!
/usr/lib/ruby/2.3.0/net/http.rb:882:in `rescue in block in connect': Failed to open TCP connection to 172.16.0.5:80 (No route to host - connect(2) for "172.16.0.5" port 80) (Errno::EHOSTUNREACH)
    from /usr/lib/ruby/2.3.0/net/http.rb:879:in `block in connect'
    from /usr/lib/ruby/2.3.0/timeout.rb:91:in `block in timeout'
    from /usr/lib/ruby/2.3.0/timeout.rb:101:in `timeout'
    from /usr/lib/ruby/2.3.0/net/http.rb:878:in `connect'
    from /usr/lib/ruby/2.3.0/net/http.rb:863:in `do_start'
    from /usr/lib/ruby/2.3.0/net/http.rb:852:in `start'
    from /usr/lib/ruby/2.3.0/net/http.rb:474:in `get_response'
    from /usr/lib/ruby/2.3.0/net/http.rb:456:in `get'
    from /home/user/bin/SecGen/projects/SecGen20200522_112949/lib/post_provision_test.rb:62:in `test_html_returned_content'
    from /home/user/bin/SecGen/projects/SecGen20200522_112949/puppet/in_the_wild/modules/moinmoin_195/secgen_test/moinmoin_195.rb:13:in `test_module'
    from /home/user/bin/SecGen/projects/SecGen20200522_112949/lib/post_provision_test.rb:34:in `run'
    from /home/user/bin/SecGen/projects/SecGen20200522_112949/puppet/in_the_wild/modules/moinmoin_195/secgen_test/moinmoin_195.rb:17:in `<main>'

Running tests for unrealirc
FAILED: Port 6667 is closed at 172.16.0.5 (in_the_wild)!
Post provision tests contained failures!

Running tests for samba
FAILED: Port 139 is closed at 172.16.0.5 (in_the_wild)!
Post provision tests contained failures!

[cliffe]

Try adding --no-tests

Which prevents post-provisioning tests from running.

[thompslu]

Does vagrant download the VM and start to provision after creation of random SecGen scenario? Could you please give me the exact command you're trying to use to start the initial scenario.

Can you test this command for me to see what output you get for this : -

ruby secgen.rb --scenario scenarios/tests/all_bases.xml run

This just gives me an idea to see if you can get base boxes running in the first instance. This just gives me a starting point to see if its configured correctly.

regards

thompslu

[0xdreadnaught]

Adding --no-tests to the flawed fortress command in the startup guide results in the following: Reading available network modules... Reading network: modules/networks/public/public_network_2 Reading network: modules/networks/public/public_network_1 Reading network: modules/networks/host_only/private_network_1 Reading network: modules/networks/host_only/private_network_2 Reading network: modules/networks/host_only/private_network_3 5 network modules loaded Resolving systems: randomising scenario... Filtered to modules matching: {"platform"=>["linux"], "type"=>["server"]} ~= (n=2) Selecting module: Debian 9 Stretch Server (modules/bases/debian_stretch_server) Module added: Debian 9 Stretch Server (modules/bases/debian_stretch_server) Filtered to modules matching: {"type"=>["flag_generator"]} ~= (n=5) Selecting module: Random 8 Character Hex Generator (modules/generators/flag/flag_8char_hex) Adding module_path to datastore (unique_encoders) to ensure same module not selected multiple times Module includes local calculation of output. Processing... Running: ruby modules/generators/flag/flag_8char_hex/secgen_local/local.rb --b64 ... /usr/lib/ruby/vendor_ruby/bundler/source/git.rb:191:in rescue in load_spec_files': http://github.com/nicanor/braille.git (at master@2c861ea) is not yet checked out. Runbundle installfirst. (Bundler::GitError) from /usr/lib/ruby/vendor_ruby/bundler/source/git.rb:188:in load_spec_files' from /usr/lib/ruby/vendor_ruby/bundler/source/path.rb:92:in local_specs' from /usr/lib/ruby/vendor_ruby/bundler/source/git.rb:159:inspecs' from /usr/lib/ruby/vendor_ruby/bundler/lazy_specification.rb:53:in __materialize__' from /usr/lib/ruby/vendor_ruby/bundler/spec_set.rb:90:inblock in materialize' from /usr/lib/ruby/vendor_ruby/bundler/spec_set.rb:87:in map!' from /usr/lib/ruby/vendor_ruby/bundler/spec_set.rb:87:inmaterialize' from /usr/lib/ruby/vendor_ruby/bundler/definition.rb:137:in specs' from /usr/lib/ruby/vendor_ruby/bundler/definition.rb:182:inspecs_for' from /usr/lib/ruby/vendor_ruby/bundler/definition.rb:171:in requested_specs' from /usr/lib/ruby/vendor_ruby/bundler/environment.rb:18:in requested_specs' from /usr/lib/ruby/vendor_ruby/bundler/runtime.rb:13:in setup' from /usr/lib/ruby/vendor_ruby/bundler.rb:92:insetup' from /usr/lib/ruby/vendor_ruby/bundler/setup.rb:18:in <top (required)>' from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:inrequire' from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require' Module failed to run (bundle exec ruby modules/generators/flag/flag_8char_hex/secgen_local/local.rb) Failed to resolve scenario. No conflicts, but failed to resolve scenario -- this is a sign there is something wrong in the config (scenario / modules) Please review the scenario -- something is wrong.

On Fri, May 22, 2020 at 7:18 PM Cliffe notifications@github.com wrote:

Try adding --no-tests

Which prevents post-provisioning tests from running.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cliffe/SecGen/issues/139#issuecomment-632940912, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHQE35PGKJBMSXIP32ZQP3RS4B4PANCNFSM4HAUMUFA .

[thompslu]

Please read my comment above just after @cliffe comment this way I can help you get a base reading for the system via the test. From the minimal output you provide here it also seems like bundle install error might be causing you a problem which suggests ruby is looking for the Braille.git

cd in to SecGen directory and bundle install again see if it throws any errors, do a bundle update.

If none of this works I would highly recommend if in an VM to revert back snapshot or if bare metal to back to square one “fresh install” and follow the installation steps it outlines on the main SecGen page precisely, without going off and doing any extra commands only change vagrant to version to 2.2.6 in the already stated command. Never use sudo commands unless stated in the installation as this can sometimes cause problems.

When you do the pull of the files to your Ubuntu box just use

git clone https:// SecGen link not sudo.

When updating Ubuntu only do sudo apt-update and apt-get upgrade that’s all I needed.

[firewallzer0]

So I went through similar steps to @DreadnaughtSec without the full upgrade option. Starting with a fresh image of Ubuntu 16.04.6

sudo apt-get install open-vm-tools -y
sudo apt-get update
sudo apt-get upgrade -y
reboot
sudo apt-get update
sudo apt-get upgrade -y
sudo apt autoremove
wget https://releases.hashicorp.com/vagrant/2.2.6/vagrant_2.2.6_x86_64.deb
sudo apt-get install ./vagrant_2.2.6_x86_64.deb
sudo apt-get install ruby-dev zlib1g-dev liblzma-dev build-essential patch virtualbox ruby-bundler imagemagick libmagickwand-dev exiftool libpq-dev libcurl4-openssl-dev libxml2-dev graphviz graphviz-dev libpcap0.8-dev git
cd /opt
sudo git clone https://github.com/cliffe/SecGen #Sudo needed for cloning into /opt
mkdir ~/bin
cp -r ./SecGen ~/bin
cd ~/bin/SecGen 
bundle install
mkdir ./projects
chmod 777 ./projects
ruby secgen.rb run

Ran into this error:

Timed out while waiting for the machine to boot. This means that
Vagrant was unable to communicate with the guest machine within
the configured ("config.vm.boot_timeout" value) time period.

If you look above, you should be able to see the error(s) that
Vagrant had when attempting to connect to the machine. These errors
are usually good hints as to what may be wrong.

If you're using a custom box, make sure that networking is properly
working and you're able to connect to the machine. It is a common
problem that networking isn't setup properly in these boxes.
Verify that authentication configurations are also setup properly,
as well.

If the box appears to be booting properly, you may want to increase
the timeout ("config.vm.boot_timeout") value.
Non-zero exit status...
Error provisioning VMs, destroying VMs and exiting SecGen.

Next I tried: ruby secgen.rb --scenario scenarios/tests/all_bases.xml run Ran into an issue that the virtual machine had entered an invalid state "paused" I was looking at the machine at the time through virtualbox gui and saw that 64bit virtualization was not supported. I have dealt with that error before with nested VMs.

What happened is that I had not enabled virtualization within the VM. So I shutdown the VM and enabled the setting "Virtualize Intel VT x/EPT or AMD-V/RVI" setting on the virtual machine (Using VMware Workstation 14) For Virtualbox this would be the VTx/AMD-V Acceleration setting

Then I reran the command ruby secgen.rb --scenario scenarios/tests/all_bases.xml run

Running post-provision tests...
VMs created.
Completed in 38m 14s

So I guess it's working? I have 6 VMs that were created and are still running.

Now when I run: ruby secgen.rb run I get a VM that was spun up successfully.

[thompslu]

Hi firewallzer0 Yes that’s correct , you have six (6)default state base boxes that are un-provisioned ready for which ever scenarios you choose to run, there may be a few tweaks here and their such as version specific software that you may need to sort but until you run the scenario you want you won’t know what exactly you need . Definitely like you say making sure the x64 virtualisation option is selected helps as the x64 box will not hang on vagrant.

By any chance did you follow the nullbyte tutorial for installation? Always ensure you check this official repo every so often for any updates, issues or installation instructions.

Regards

thomplsu

[kiddy818]

priority_queue.c:820:4: note: in expansion of macro ‘rb_funcall’ rb_funcall(Qnil, id_format, 4, rb_str_new2("NODE%i -> NODE%i;\n"), the 3rd value was wrong, it should be 3 （change 4 ->3）

[kbs-code]

I had the same issue as @teminvictus while installing today on MX Linux 19.3. Removing huffman from the Gemfile did provide a workaround.

[fgosew]

Here's a modified Gemfile fixing all the issues with current Debian based Ruby installations (tested with Kali Linux 2021.1 and Ubuntu 20.04). I created an override for all packages responsible for the build failing. As I'm no Ruby developer that might break a thing or two because I took the liberty to not rename the packages failing but instead forking them and modifying what was missing or broken. Not sure if this is the code of conduct with Ruby though ;).

index ff1ed012..2de89e01 100644
--- a/Gemfile
+++ b/Gemfile
@@ -23,12 +23,13 @@ gem 'process_helper'
 gem 'ovirt-engine-sdk'
 gem 'duplicate'
 gem 'smbhash'
-gem 'digest-whirlpool'
+gem 'digest-whirlpool', :git => "https://github.com/fgosew/ruby-digest-whirlpool"
 gem 'digest-siphash'
 gem 'scrypt'
 gem 'braille', :git => "http://github.com/nicanor/braille.git"
 gem 'bases'
-gem 'huffman'
+gem 'priority_queue', :git => "https://github.com/fgosew/priority_queue"
+gem 'huffman', :git => "https://github.com/fgosew/huffman"
 gem 'ruby-graphviz'
 gem 'rsa'
 gem 'gpgmeh'

[cliffe]

Here's a modified Gemfile fixing all the issues with current Debian based Ruby installations (tested with Kali Linux 2021.1 and Ubuntu 20.04). I created an override for all packages responsible for the build failing. As I'm no Ruby developer that might break a thing or two because I took the liberty to not rename the packages failing but instead forking them and modifying what was missing or broken. Not sure if this is the code of conduct with Ruby though ;).

index ff1ed012..2de89e01 100644
--- a/Gemfile
+++ b/Gemfile
@@ -23,12 +23,13 @@ gem 'process_helper'
 gem 'ovirt-engine-sdk'
 gem 'duplicate'
 gem 'smbhash'
-gem 'digest-whirlpool'
+gem 'digest-whirlpool', :git => "https://github.com/fgosew/ruby-digest-whirlpool"
 gem 'digest-siphash'
 gem 'scrypt'
 gem 'braille', :git => "http://github.com/nicanor/braille.git"
 gem 'bases'
-gem 'huffman'
+gem 'priority_queue', :git => "https://github.com/fgosew/priority_queue"
+gem 'huffman', :git => "https://github.com/fgosew/huffman"
 gem 'ruby-graphviz'
 gem 'rsa'
 gem 'gpgmeh'

That's great! Ideally you would also submit pull requests to upstream to work towards getting the issues resolved in the original gems. Thanks for your efforts!