remove-dual.sh doesn't work with crontab

[cybermonkee]

When executing the script using crontab I got the following std-err out: remove-dual-or.sh: line 6: ipset: command not found compare-dual2.sh: line 3: ipset: command not found

This error was fixed by putting in the absolute path to ipset in the script like so:

!/bin/bash

set -x

./compare-dual2.sh > remove-dual2 for i in cat remove-dual2 ; do /usr/sbin/ipset del tor2-ddos $i done;

I set a crontab for every minute to run these remove scripts - for the crontab to work properly, the *.sh files needed to be installed in the home path of the crontab user.

[Enkidu-6]

Yes, those files were originally meant to be run manually and I didn't fine tune them for cron jobs. I assumed users interested in automating them will make their own modifications. Nevertheless I have made some changes to the files for the new version and they are still being tested. I combined some files and reduced the number of individual scripts and I also created a new directory with a few files meant for cron jobs in the cron directory.

You can find them in the test branch of this repository. Feel free to give them a try and let me know. The compare file now also gives you an option to delete IPs from the ipset.

Also running them every minute is a bit too much IMHO. Remember, each time you're running the script you're pulling a list of all relays and regardless of what source you're using, sooner or later you'll hit some kind of an API limit. Those lists are generally updated once an hour or every 30 minutes and pulling them every minute is a bit unnecessary. Not to mention that at least I'm not getting many dual-OR relays in the list at such a high rate to justify removing them every minute.

[cybermonkee]

I hadn't thought about their being an API limitation TBH. But I was getting quite a few dual hits on the relay.

I think it useful that the compare and remove scripts are separate. What I do is set up a watch something like:

watch -n 1800 ls -la *.log

That looks out for log files generated by cron, I know then if the files contain 0 bytes or not as to whether there are items in the dual hit list.

I have created a variant of compare-dual.sh that doesn't download the list of relays, this allows me to do a compare without pulling the list.

[Enkidu-6]

I modified the files in the cron directory of the test branch. You're welcome to give them a try. What they do is first they check to see if the file exist, if it does, the script uses the existing file and if not, pulls the list. It also checks if the existing list is older than one hour, if so, it pulls a new one and if not, it uses the existing list in /var/tmp. This way, you can run the cron as often as you wish.

The relay sources are generated only once an hour at onionoo.torproject.org so pulling them more than once an hour gets a duplicate anyway.

I'm also pulling the lists from my other repository on github. https://github.com/Enkidu-6/tor-relay-lists . I'm generating and updating the lists twice an hour and sending them to the repository.