[New analyzer] Spyse.com API

[eshaan7]

Mark from their marketing team reached out to me on Twitter for this integration a long time back but I forgot about it.

SDK: https://github.com/spyse-com/spyse-python

[Palash-Vishnani]

I can work on this issue. Will you please elaborate what do I have to do exactly.?

[sp35]

Hi @Palash-Vishnani, thank you for your interest in contributing to the project. You need to look for the services which can be integrated as an analyzer.

• https://spyse-dev.readme.io/reference/ips#ip_details (observable_type - ip)
• https://spyse-dev.readme.io/reference/emails#email_details (observable_type - general)
• https://spyse-dev.readme.io/reference/domains#domain_details (observable_type - domain)
• https://spyse-dev.readme.io/reference/history#domain_whois_history (observable_type - domain)

See the docs on How to add new analyzer? Join us on Slack @Honeynet Slack, if you need any help regarding development.

[m0mosenpai]

Another thing @Palash-Vishnani, we are in works of releasing a new version which brings about a lot of changes in the project (It may or may not be of concern to you depending on the timeline of your PR). The process for the addition of analyzer is not going to be affected as much but just giving a heads up.

[eshaan7]

Hello, @Palash-Vishnani. Welcome to IntelOwl, you are more than welcome to work on this issue.

Here are some tips:

• Fork the repository and checkout to develop-2 branch, from there create a new branch dev-spyse

  git checkout develop-2
  git checkout -b dev-spyse

• The docs on How to add new analyzer? should help you but as @m0mosenpai mentioned, we are currently changing a lot of stuff and that means the docs are not up-to-date. That's why I try to explain below with some additional steps.
• The analyzer configurations are defined in the configuration/analyzer_config.json file. Similar to the existing entries, you've to create a new entry for Spyse.
• Let me give you an example: If you look at the Shodan_Search entry in this JSON file, you will see it's python_module attribute points to a file named shodan.py inside which we have a class named Shodan which is a child class of ObservableAnalyzer class. Similar to this, you should create a spyse.py file with a class Spyse that inherits from ObservableAnalyzer. You will overwrite the set_params method with the logic of getting the API key and the run method is where the main logic lives.
• As @sp35 has mentioned the various different endpoints that Spyse has so we want to support all of them but that doesn't mean we need different analyzer/.py file for each. Instead, it can be made as part of a single new analyzer only. Meaning the observable_supported of this new analyzer will be: ["ip", "domain", "generic"]. Depending on which observable classification was passed to the class instance (accessed with: self.observable_classification), the endpoint to be queried will change. (Similar kind of logic is in the threatminer.Threatminer.run method so you can take inspiration from that.)
• Once you have a new entry in the JSON file which points to a new .py file, you can test your analyzer by opening the GUI on http://localhost/ and creating a new scan. (Remember to create a django superuser using the command line first)
• If the analyzer is working fine, now you need to write tests for it. We've made this very simple so all you need to do is overwrite the _monkeypatch method of the Spyse class. On how to implement this function, you can take inspiration from the various analyzers that are there in the api_app/analyzers_manager/observable_analyzers directory.

[eshaan7]

Hi, @Palash-Vishnani. Feel free to drop into our slack @Honeynet Slack, if you need any help regarding development.

[Palash-Vishnani]

Sorry for the delay, I was busy because of my exams. I will start working on this issue soon now.

[eshaan7]

Update:

Use the develop branch, the develop-2 has been deprecated.

[Palash-Vishnani]

@Eshaan7 there are some things in which I am still confused:

• What description should I write in the new Spyse analyzer in configuration/analyzer_config.json file?
• What will be the value of base_url, and also can you tell me exactly how should I overwrite the set_params method with the logic of getting the API key inside Spyse class in spyse.py?
• I followed your instructions step-by-step but got stuck in the last three points, can you please explain the last three steps in detail.
• Lasty, @sp35 said something about looking for the services which can be integrated as an analyzer for the three different observable_type, please tell me in detail about that too.

I am a beginner that's why I am facing such problems, but I'll definitely solve this issue if I'll get to know what exactly do I have to do further.

[m0mosenpai]

@Palash-Vishnani

• Description could be a one liner describing what the analyzer does. Ex; Scans files/IPs for X using Y. (similar to how it's done in the other analyzers)
• base_url should hold the base endpoint for the API. Example: here the part https://api.spyse.com/v4/data/ is constant in all endpoints (i think) so this would be the base_url.

However, you won't need to mess with this since as mentioned in the first comment on the issue, you can use the official Python SDK. You can simply import that package and use your API token to make requests using that. (https://github.com/spyse-com/spyse-python) (check their docs for more info on the usage)

• Each analyzer we integrate either scans files or observables - observable being IPs/Domains/URLs/Hashes. This analyzer supports IPs and Domains (not sure if there are more, you'll have to check their API Docs). Based on the value of self.observable_classification , you'll have to query a different endpoint. Ex: The above screenshot shows an endpoint for IPs. Similarly, there's another for Domains. Something like an if condition could help you decide where you want to query for data depending on the value of the variable. Do see other implementations of analyzers once.

• Once you have created the analyzer, analyzers_config.json stores it's configuration info (details of which are there in the docs). Once you add that, you have to build and launch the Django Backend so you can test your changes. localhost is where you will have to go in order to access the GUI and test if the analyzers runs. For you to access the GUI, you'll need to create a superuser account. (Run the backend, open another terminal window/tab and run

  docker exec -ti intelowl_uwsgi python3 manage.py createsuperuser

  See https://intelowl.readthedocs.io/en/develop/Tests.html for details on testing.

• For this point, it can be a little confusing at first so I would encourage you to reach till here and submit a draft PR so we can see your progress and help you more effectively. Try and go through other analyzers to see how _monkeypatch is implemented and see if you can replicate it otherwise.

• Also, @sp35 was referring to different endpoint based on value of self.observable_classification, similar to what @Eshaan7 was saying.

[Palash-Vishnani]

Hey, just confirming ..which git command should I use to push this new dev-spyse branch from my local system to my remote forked IntelOwl repository??? And the origin should be master or develop branch?