Docker configuration persistence errors

[ehsandeep]

What's the problem (or question)?

Config file error while running the binary even config file placed under $GOPATH/bin

Do you have an idea for a solution?

Not sure!

How can we reproduce the issue?

1. Run the subfinder from anywhere in root path

What are the running context details?

• Installation method (go get):
• Client OS (Ubuntu)

root@vps:~# ls $GOPATH/bin | grep config
config.json
root@vps:~# subfinder 

             __     ___ __          __            
.-----.--.--|  |--.'  _|__.-----.--|  .-----.----.
|__ --|  |  |  _  |   _|  |     |  _  |  -__|   _|
|_____|_____|_____|__| |__|__|__|_____|_____|__|  

SubFinder v0.1.0          Made with ❤ by @Ice3man
==================================================open ./config.json: no such file or directory
root@vps:~# 

[Ice3man543]

Hi @madaratech I am having trouble reproducing your issue. Can you provide me with a directory listing of the files. What is the name of your configuration file ?

Are you running the tool from the $GOPATH/bin directory or you have copied the tool to your root directory. If so, you have to copy the config.json file to any dir you have the tool in. Or, you could create a symbolic link to the subfinder tool in a directory with it's config file. Thanks.

[ehsandeep]

If i'm running the tool from

root@vps:~/go/src/github.com/ice3man543/subfinder#

it's working well, as you suggested in readme to copy config.json in /go/bin after doing that, if i'm trying to run the subfinder from root dir or anywhere else, i'm getting that error.

[Ice3man543]

Hi @madaratech . You need to have both subfinder and config.json in the same directory. Can you provide a directory listing of where you want to run the tool ?

[ehsandeep]

Got it clear now, I need to have the config file in dir wherever I want to use it.

Thanks!

[tomnomnom]

I realise this issue is closed, but I've seen the same thing.

The README specifies there should be a config.json file in $GOPATH/bin/, or whichever directory the tool is in. This is not the case. The config.json file must be in the current working directory:

tom@bash:~▶ go get github.com/ice3man543/subfinder
tom@bash:~▶ subfinder -h
open ./config.json: no such file or directory
tom@bash:~▶ echo '{}' > $GOPATH/bin/config.json
tom@bash:~▶ subfinder -h
open ./config.json: no such file or directory
tom@bash:~▶ echo '{}' > ./config.json
tom@bash:~▶ subfinder -h
Usage of subfinder:
  -b    Use bruteforcing to find subdomains
  -c    Use colour in outpout (default true)
  -d string
        Domain to find subdomains for
...

This means the user must have a config.json file in every directory they wish to use the tool in.

Given that the tool operates OK with only an empty object in the JSON file, I would suggest making it so that the tool still works when there is no config.json file found.

[Ice3man543]

Hi @tomnomnom, Thanks for commenting. I have modified the tool to read the config.json file from the path the tool is placed in and also not to raise any error if it is not found. Is that Ok? Or do i need to consider adding the configuration file in ~/.config or something. I don't know what's the right way to do this in golang. Thanks.

[tomnomnom]

@Ice3man543 thanks for the super fast response :)

Personally I think having it in $GOPATH/bin (or wherever else the binary is located) is a little strange / not really the place for it. Something under ~/.config/ would make more sense IMO as it's where many tools store their user-specific (as apposed to system-wide) configuration files.

[Ice3man543]

@tomnomnom So I should create a directory under ~/.config where the configuration file should be stored. But wouldn't it require more effort on user's part. If i do that, i would have to create an installer script. Or we could simply let it remain there. Any ideas how to create an installer script for golang tool.

This way, we could just put it in the tool's dir like /opt/subfinder/config.json

Also, do you think adding multiple archives to find subdomains is a good idea. I am currently adding support for Wayback Machine. Do you think it would be better to add support for others too? Also, any sources you can recommend.

Thanks.

[tomnomnom]

IMO the tool could/should just create the directory and file if it doesn't exist when you run it. No need for an install script or anything like that; which is fairly standard behaviour for linux tools that require config etc.

massren is an example of such a tool written in Go that does that. Here's where it handles some of that stuff as an example.

Generally I think as many sources as possible is a good idea! I don't think I know of any you don't already have though sorry!

[Ice3man543]

Hey @tomnomnom @madaratech I have edited the code to add the ability to have config.json in $HOME/.config/subfinder directory. The directory is automatically created if it doesn't exists when the tool is run for the first time. I guess it fixes the issue. Check it out. Another thing, please clone the whole thing again since due to some bad commits, had to force push. :+1:

[Ice3man543]

Hi guys @tomnomnom @madaratech, I have updated the code to have it store values in $HOME/.config/subfinder/config.json file. The user can set these values using --set-config option. I think this resolves this issue. If so, please close it.

[ehsandeep]

Thank you @Ice3man543

[ehsandeep]

Hey @Ice3man543,

root@vps:~/subfinder# docker run -it subfinder --set-config securitytrailsKey=zSVWtFQRR4DamhVUyzWKy*************

[NOTE] Edit /root/.config/subfinder/config.json with your options !
             __     ___ __          __            
.-----.--.--|  |--.'  _|__.-----.--|  .-----.----.
|__ --|  |  |  _  |   _|  |     |  _  |  -__|   _|
|_____|_____|_____|__| |__|__|__|_____|_____|__|  

SubFinder v0.1.0          Made with ❤ by @Ice3man
==================================================
panic: reflect: call of reflect.Value.SetString on zero Value

goroutine 1 [running]:
reflect.flag.mustBeAssignable(0x0)
        /usr/local/go/src/reflect/value.go:221 +0xe6
reflect.Value.SetString(0x0, 0x0, 0x0, 0x7ffd62782ef7, 0x20)
        /usr/local/go/src/reflect/value.go:1535 +0x2b
main.main()
        /go/src/github.com/ice3man543/subfinder/main.go:87 +0x3a6

added both keys using set-config option but for SecurityTrails got following error.

[Ice3man543]

Hey @madaratech can you tell me what steps you followed? If you have passed options on first run, then the error is likely. Else, there seems to be some problem. Am. I right?

[Ice3man543]

Yup. I know what the problem is. You are running the tool for the first time with --set-config flag. That will cause an error since the config file hasn't been read yet. What you should do is run the tool once and then run it again specifying --set-config option. The first time will initialize it and then you can have fun with it. Correct me if I am wrong !

[ehsandeep]

No, I'm not running it for the 1st time, did 2-3 scan without keys and tried to add that later!

infact able to add those 2 other keys, having problem with SecurityTrails only.

[Ice3man543]

@madaratech Sorry. It seems that the docker image can't hold config.json permanently. Looks Like i need to learn about docker

[Ice3man543]

Do you know any solution to persist data across docker runs ?

[Ice3man543]

I don't have any errors while adding SecurityTrails key outside docker. Try to do it without the docker image!

[Ice3man543]

@madaratech Can you run it with Virustotal option ?

[ehsandeep]

yeah, as I said before I'm able to add VirusTotal and PassiveDNS key, will try to run the same outside the docker and let you know.

[Ice3man543]

@madaratech I read about docker for some time. The solution I have found is quite simple. You can run the binary in your host system and let it create the directory structure of files, or you can run mkdir $HOME/.config/subfinder and then copy the config.json file in that directory and edit it with values. After that, you can pass it as a volume using the following sample command. sudo docker run -v /home/ice3man/.config/subfinder:/root/.config/subfinder -it subfinder -d freelancer.com Is it appropirate @madaratech ?

[tamjid0x01]

using this command in kali: GO111MODULE=on go get -v github.com/projectdiscovery/subfinder/cmd/subfinder

It's work properly 👍