Windows: Error in line 707

ghost commented 6 years ago

Running 0.7.3 in windows line 707 gets an error. It appears Pulledpork is now using grep a Unix command that is not compatible with Windows. :(

Prepping rules from opensource.gz for work.... Done! Reading rules... Can't find Unicode property definition "u" in regex; marked by <-- HERE in m/^d:\winids\pu <-- HERE lledpork\temp/tha_rules/.$/ at d:\winids\pulledpork\pulledpork.pl line 707.

shirkdog commented 6 years ago

can you try master? it has been bumped to version 0.7.4. and those line numbers do not match up.

ghost commented 6 years ago

Ok, I tried it again with same results.

C:\Users\Operator>perl d:\winids\pulledpork\pulledpork.pl -c d:\winids\pulledpor

k\etc\pulledpork.conf -T

https://github.com/shirkdog/pulledpork

  _____ ____

 `----,\    )

  `--==\\  /    PulledPork v0.7.4 - Helping you protect your bitcoin wallet!

   `--==\\/

 .-~~~~-.Y|\\_  Copyright (C) 2009-2017 JJ Cummings, Michael Shirk

@/ / 66\ and the PulledPork Team!

|    \   \   _(")

 \   /-| ||'--'  Rules give me wings!

  \_\  \_\\



'uname' is not recognized as an internal or external command, operable program or batch file.

Checking latest MD5 for snortrules-snapshot-29110.tar.gz....

Rules tarball download of snortrules-snapshot-29110.tar.gz....

        They Match

        Done!

IP Blacklist download of https://talosintelligence.com/documents/ip-blacklist....

Reading IP List...

Checking latest MD5 for opensource.gz....

Rules tarball download of opensource.gz....

        They Match

        Done!

Prepping rules from snortrules-snapshot-29110.tar.gz for work....

No such file in archive: 'doc/signatures/rules/VRT-License.txt' at d:\winids\pulledpork\pulledpork.pl line 363.

Could not find an entry for 'doc/signatures/rules/VRT-License.txt' at d:\winids\pulledpork\pulledpork.pl line 363.

        Done!

Prepping rules from opensource.gz for work....

        Done!

Reading rules...

Can't find Unicode property definition "u" in regex; marked by <-- HERE in m/^d:\winids\pu <-- HERE lledpork\temp/tha_rules/.$/ at d:\winids\pulledpork\pulledpork.pl line 719.

C:\Users\Operator>

Best regards,

Michael...

From: Shirkdog [mailto:notifications@github.com] 
Sent: Monday, December 11, 2017 6:45 AM
To: shirkdog/pulledpork <pulledpork@noreply.github.com>
Cc: mesteele101 <michaels@go2dds.com>; Author <author@noreply.github.com>
Subject: Re: [shirkdog/pulledpork] Windows: Error in line 707 (#299)

can you try master? it has been bumped to version 0.7.4. and those line numbers do not match up.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub <https://github.com/shirkdog/pulledpork/issues/299#issuecomment-350701440> , or mute the thread <https://github.com/notifications/unsubscribe-auth/ABNoev_3Z_isepjWXrcAD0jAxkIseYtsks5s_RWmgaJpZM4Q72l_> .  <https://github.com/notifications/beacon/ABNoepmuQ-lRxQJZkh0lb6mwBUcqLO-Pks5s_RWmgaJpZM4Q72l_.gif>

ghost commented 6 years ago

I tried using –nPT to bypass the opensource file and it appears to work but crashes on Windows getting past that.

PulledPork crashes on Windows at the same place (line 719) no matter what switches are specified L

C:\Users\Operator>perl d:\winids\pulledpork\pulledpork.pl -c d:\winids\pulledpork\etc\pulledpork.conf -nPT

https://github.com/shirkdog/pulledpork

  _____ ____

 `----,\    )

  `--==\\  /    PulledPork v0.7.4 - Helping you protect your bitcoin wallet!

   `--==\\/

 .-~~~~-.Y|\\_  Copyright (C) 2009-2017 JJ Cummings, Michael Shirk

@/ / 66\ and the PulledPork Team!

|    \   \   _(")

 \   /-| ||'--'  Rules give me wings!

  \_\  \_\\



'uname' is not recognized as an internal or external command, operable program or batch file.

Prepping rules from snortrules-snapshot-29110.tar.gz for work....

No such file in archive: 'doc/signatures/rules/VRT-License.txt' at d:\winids\pulledpork\pulledpork.pl line 363.

Could not find an entry for 'doc/signatures/rules/VRT-License.txt' at d:\winids\pulledpork\pulledpork.pl line 363.

        Done!

Skipping opensource.gz as -nPT was specified

        Done!

Reading rules...

Can't find Unicode property definition "u" in regex; marked by <-- HERE in m/^d:\winids\pu <-- HERE lledpork\temp/tha_rules/.$/ at d:\winids\pulledpork\pulledpork.pl line 719.

C:\Users\Operator>

Best regards,

Michael...

From: Shirkdog [mailto:notifications@github.com] 
Sent: Monday, December 11, 2017 6:45 AM
To: shirkdog/pulledpork <pulledpork@noreply.github.com>
Cc: mesteele101 <michaels@go2dds.com>; Author <author@noreply.github.com>
Subject: Re: [shirkdog/pulledpork] Windows: Error in line 707 (#299)

can you try master? it has been bumped to version 0.7.4. and those line numbers do not match up.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub <https://github.com/shirkdog/pulledpork/issues/299#issuecomment-350701440> , or mute the thread <https://github.com/notifications/unsubscribe-auth/ABNoev_3Z_isepjWXrcAD0jAxkIseYtsks5s_RWmgaJpZM4Q72l_> .  <https://github.com/notifications/beacon/ABNoepmuQ-lRxQJZkh0lb6mwBUcqLO-Pks5s_RWmgaJpZM4Q72l_.gif>

shirkdog commented 6 years ago

So these changes were brought in with perltidy to clean up all the perlcode...nothing was changed in the code.

Try this patch to your current 0.7.4 master, and if this works...there is something wrong with ActiveState perl on Windows. Whitespace should never break a perl script.

fix.patch.gz

shirkdog commented 6 years ago

The other thing that might have happened with perltidy, is UNIX vs. Windows linebreaks.

If the patch still fails, but the attached windows.pl works, then your issue is you have to run unix2dos on this script before you use it.

windows.pl.gz

ghost commented 6 years ago

Still has a problem with line:

Reading rules...

Can't find Unicode property definition "u" in regex; marked by <-- HERE in m/^d:\winids\pu <-- HERE lledpork\temp/tha_rules/.$/ at d:\winids\pulledpork\pulledpork.pl line 719.

Best regards,

Michael...

From: Shirkdog [mailto:notifications@github.com] Sent: Monday, December 11, 2017 7:47 PM To: shirkdog/pulledpork pulledpork@noreply.github.com Cc: mesteele101 michaels@go2dds.com; Author author@noreply.github.com Subject: Re: [shirkdog/pulledpork] Windows: Error in line 707 (#299)

So these changes were brought in with perltidy to clean up all the perlcode...nothing was changed in the code.

Try this patch to your current 0.7.4 master, and if this works...there is something wrong with ActiveState perl on Windows. Whitespace should never break a perl script.

fix.patch.gz https://github.com/shirkdog/pulledpork/files/1549883/fix.patch.gz

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/shirkdog/pulledpork/issues/299#issuecomment-350907699 , or mute the thread https://github.com/notifications/unsubscribe-auth/ABNoeg8OlsX0gGWbs_tKo2GdZY2X8efxks5s_c0RgaJpZM4Q72l_ . https://github.com/notifications/beacon/ABNoeoxlBTJWI21Fw_T2WkoD-1eUQpDpks5s_c0RgaJpZM4Q72l_.gif

ghost commented 6 years ago

It now has a problem with like 722, which is the same line 719 from the original pulledpork.pl and windows.pl.

Reading rules...

Can't find Unicode property definition "u" in regex; marked by <-- HERE in m/^d:\winids\pu <-- HERE lledpork\temp/tha_rules/.$/ at d:\winids\pulledpork\pulledpork.pl line 722.

Best regards,

Michael...

From: Shirkdog [mailto:notifications@github.com] Sent: Monday, December 11, 2017 9:08 PM To: shirkdog/pulledpork pulledpork@noreply.github.com Cc: mesteele101 michaels@go2dds.com; Author author@noreply.github.com Subject: Re: [shirkdog/pulledpork] Windows: Error in line 707 (#299)

The other thing that might have happened with perltidy, is UNIX vs. Windows linebreaks.

If the patch still fails, but the attached windows.pl works, then your issue is you have to run unix2dos on this script before you use it.

windows.pl.gz https://github.com/shirkdog/pulledpork/files/1550021/windows.pl.gz

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/shirkdog/pulledpork/issues/299#issuecomment-350921809 , or mute the thread https://github.com/notifications/unsubscribe-auth/ABNoeivgutajoLyyPoR6lzxQKqrwDqi6ks5s_d_-gaJpZM4Q72l_ . https://github.com/notifications/beacon/ABNoehZfT6SkRMHXqDddFfT7cDti89E0ks5s_d_-gaJpZM4Q72l_.gif

ghost commented 6 years ago

Running the pulledpork.pl patch you just sent gets the same error. The patch looked exactly like the original code?

Reading rules... Can't find Unicode property definition "u" in regex; marked by <-- HERE in m/^d:\winids\pu <-- HERE lledpork\temp/tha_rules/.$/ at d:\winids\pulledpork\pulledpork.pl line 719.

Running the windows.pl now has a problem with line 722, which is the same code on line 719 from the original pulledpork.pl mentioned in the error above,

Reading rules... Can't find Unicode property definition "u" in regex; marked by <-- HERE in m/^d:\winids\pu <-- HERE lledpork\temp/tha_rules/.$/ at d:\winids\pulledpork\pulledpork.pl line 722.

ghost commented 6 years ago

The code changed as shown below. If I replace the code from 0.7.2 into 0.7.4 Pulledpork works in windows. Line 718 in 0.7.2 had to be changed or it faulted.

I'm not sure on line 719 which is the line that faults in 0.7.4. It might be 'grep' is not compatible with Windows, or is it the '/^$path$file$/' and the characters that it produces, or both?

--Code from 0.7.4 717 while (defined($file = readdir DIR)) { 718 next 719 if grep /^$path$file$/, 720 @local_rules; #don't read local rule files 721 open(DATA, "$path$file") || croak "Couldn't read $file - $!\n"; 722 @elements = ;

-- Code from 0.7.2 717 while (defined($file = readdir DIR)) { 718 open( DATA, "$path$file" ); # || croak "Couldn't read $file - $!\n"; 719 @elements = ;

shirkdog commented 6 years ago

I added some OS detection from perl, so we should be able to add a workaround for that code for Windows only, grep is not "UNIX" grep, but is a perl function.

http://perldoc.perl.org/functions/grep.html

ghost commented 6 years ago

So is GREP something I need to add to Strawberry Perl?

How do I test if Grep is available in Perl?

This is the path: D:\winids\snort\rules

Attached is the file. Could the problem be with the path or the file?

Best regards,

Michael...

From: Shirkdog [mailto:notifications@github.com] Sent: Thursday, December 21, 2017 10:39 AM To: shirkdog/pulledpork pulledpork@noreply.github.com Cc: mesteele101 michaels@go2dds.com; Author author@noreply.github.com Subject: Re: [shirkdog/pulledpork] Windows: Error in line 707 (#299)

I added some OS detection from perl, so we should be able to add a workaround for that code for Windows only, grep is not "UNIX" grep, but is a perl function.

http://perldoc.perl.org/functions/grep.html

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/shirkdog/pulledpork/issues/299#issuecomment-353382033 , or mute the thread https://github.com/notifications/unsubscribe-auth/ABNoev0jzAtwBnzpsX7fr-FOAjVlyLLSks5tCnuogaJpZM4Q72l_ . https://github.com/notifications/beacon/ABNoet90el2Y1sBtklFj2MhjxK5GFJVkks5tCnuogaJpZM4Q72l_.gif

shirkdog commented 6 years ago

#!/usr/bin/env perl
my @bar = ("Testing", "#Comment", "Other", "Testing");
my @foo = grep(!/^#/, @bar);    # weed out comments
for (my $i=0; $i <= $#foo; $i++) {
     print "$i in foo is $foo[$i]\n";
}

This is an example from the documents. grep runs on every element in the @bar array, and only returns the strings that do not match # appearing at the beginning of the string. This is similar to what is going on in pulledpork.

This code should run without error, and if it doesn't, the only fix is to skip that syntax for Perl running on Windows.

ghost commented 6 years ago

I was able to run it on Windows:

D:>test.pl 0 in foo is Testing 1 in foo is Other 2 in foo is Testing

D:>

jrmain commented 6 years ago

I'm seeing the same problem on a Windows 8.1 system. I don't know when it started happening, I just noticed recently that when my scheduled signature update process runs, nothing is being changed.

When I run PulledPork manually, I get the exact same error as the original reporter, except that the problem path is on drive F.

Note that since this is a Windows system, paths in pulledpork.conf are specified using backslashes instead of slashes. For example, the path that seems to be at the center of this issue is specified as temp_path=f:\winids\pulledpork\temp.

Is it possible that perltidy changed the code in question (if grep /^$path$file$/,) to a form that doesn't work properly with Windows-style paths, or with Windows generally?

To test that theory, I reverted the code in 0.7.3 to the form it had in 0.7.2, changing:

next
    if grep /^$path$file$/,
    @local_rules;    #don't read local rule files

to: next if grep /^$path$file$/, @local_rules; #don't read local rule files

... and the error no longer occurs.

Wakizashimaster commented 6 years ago

As mentioned earlier in the thread change this line 709: open(DATA, "$path$file") || croak "Couldn't read $file - $!\n";

To this open(DATA, "$path$file"); # || croak "Couldn't read $file - $!\n";

It was also a suggested edit a long time ago in blog: http://www.marshalgraham.com/2011/10/install-and-configure-snort-ids-on.html

I had the same issues and commenting out the croak let me complete the downloads.

shirkdog commented 4 years ago

I believe both issues are now resolved, reopen if not the case.

ghost commented 4 years ago

I just ran the latest version on Windows and got:

C:\Windows\system32>perl d:\winids\pulledpork\pulledpork.pl -c d:\winids\pulledpork\etc\pulledpork.conf -T

    https://github.com/shirkdog/pulledpork
      _____ ____
     `----,\    )
      `--==\\  /    PulledPork v0.7.4 - Helping you protect your bitcoin wallet!
       `--==\\/
     .-~~~~-.Y|\\_  Copyright (C) 2009-2017 JJ Cummings, Michael Shirk
  @_/        /  66\_  and the PulledPork Team!
    |    \   \   _(")
     \   /-| ||'--'  Rules give me wings!
      \_\  \_\\
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

'uname' is not recognized as an internal or external command,
operable program or batch file.
Checking latest MD5 for snortrules-snapshot-29161.tar.gz....
Rules tarball download of snortrules-snapshot-29161.tar.gz....
        They Match
        Done!
IP Blacklist download of https://talosintelligence.com/documents/ip-blacklist....
Reading IP List...
Prepping rules from snortrules-snapshot-29161.tar.gz for work....
        Done!
Reading rules...
Can't find Unicode property definition "u" in regex; marked by <-- HERE in m/^d:\winids\pu <-- HERE lledpork\temp/tha_rules/.$/ at d:\winids\pulledpork\pulledpork.pl line 705.

C:\Windows\system32>

I have to Change the code from:

    if (-d $path) {
        opendir(DIR, "$path");
        while (defined($file = readdir DIR)) {

            #don't read local rule files
            next if grep /^$path$file$/, @local_rules;
            open(DATA, "$path$file") or die "Couldn't read $file - $!\n";
            @elements = <DATA>;
            close(DATA);

To This:

    if (-d $path) {
        opendir(DIR, "$path");
        while (defined($file = readdir DIR)) {
            open( DATA, "$path$file" ); # || croak "Couldn't read $file - $!\n";
            @elements = <DATA>;
            close(DATA);

Once I make the changes above, I get:

C:\Windows\system32>perl d:\winids\pulledpork\pulledpork.pl -c d:\winids\pulledpork\etc\pulledpork.conf -T

    https://github.com/shirkdog/pulledpork
      _____ ____
     `----,\    )
      `--==\\  /    PulledPork v0.7.4 - Helping you protect your bitcoin wallet!
       `--==\\/
     .-~~~~-.Y|\\_  Copyright (C) 2009-2017 JJ Cummings, Michael Shirk
  @_/        /  66\_  and the PulledPork Team!
    |    \   \   _(")
     \   /-| ||'--'  Rules give me wings!
      \_\  \_\\
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

'uname' is not recognized as an internal or external command,
operable program or batch file.
Checking latest MD5 for snortrules-snapshot-29161.tar.gz....
Rules tarball download of snortrules-snapshot-29161.tar.gz....
        They Match
        Done!
IP Blacklist download of https://talosintelligence.com/documents/ip-blacklist....
Reading IP List...
Prepping rules from snortrules-snapshot-29161.tar.gz for work....
        Done!
Reading rules...
readline() on closed filehandle DATA at d:\winids\pulledpork\pulledpork.pl line 704.
readline() on closed filehandle DATA at d:\winids\pulledpork\pulledpork.pl line 704.
readline() on closed filehandle DATA at d:\winids\pulledpork\pulledpork.pl line 704.
Reading rules...
Activating security rulesets....
        Done
Modifying Sids....
        Done!
Processing d:\winids\pulledpork\etc\enablesid.conf....
        Modified 0 rules
        Skipped 0 rules (already disabled)
        Done
Processing d:\winids\pulledpork\etc\dropsid.conf....
        Modified 0 rules
        Skipped 0 rules (already disabled)
        Done
Processing d:\winids\pulledpork\etc\disablesid.conf....
        Modified 0 rules
        Skipped 0 rules (already disabled)
        Done
Setting Flowbit State....
        Enabled 559 flowbits
        Done
Writing d:\winids\snort\rules\winids.rules....
        Done
Generating sid-msg.map....
        Done
Writing v1 d:\winids\snort\etc\sid-msg.map....
        Done
Writing d:\winids\snort\log\sid_changes.log....
        Done
Rule Stats...
        New:-------63
        Deleted:---1
        Enabled Rules:----17369
        Dropped Rules:----0
        Disabled Rules:---22469
        Total Rules:------39838
No IP Blacklist Changes

Done
Please review d:\winids\snort\log\sid_changes.log for additional details
Fly Piggy Fly!

C:\Windows\system32>

shirkdog / pulledpork

Windows: Error in line 707 #299