search

center-for-threat-informed-defense / tram

TRAM is an open-source platform designed to advance research into automating the mapping of cyber threat intelligence reports to MITRE ATT&CK®.
https://ctid.mitre-engenuity.org/our-work/tram/
Apache License 2.0
423 stars 90 forks source link

unable to upload a document #183

Closed pshurie closed 1 year ago

pshurie commented 1 year ago

below is my call along with the results: (returning 301 and not uploading new report)

curl -X POST -H 'Cookie: csrftoken=7o0DcuWyLp1doaDEgGtld5GWHFnPdOWeGqjSqeYWJWOVRK2ngm1JjbwAYlUnqEvs;sessionid=ds3gxtx2ecm6ovwuzinzm4ukytwmossj' -d '{"csrfmiddlewaretoken":"7o0DcuWyLp1doaDEgGtld5GWHFnPdOWeGqjSqeYWJWOVRK2ngm1JjbwAYlUnqEvs","name":"patTest","filename":"patText.html","type":"text/html","data":"<!DOCTYPE html>This is the content"}' http://localhost:8000/upload -v

Note: Unnecessary use of -X or --request, POST is already inferred.

mehaase commented 1 year ago

Hi @pshurie based on the response it looks like its asking for an extra slash at the end of the URL, e.g. change http://localhost:8000/upload to http://localhost:8000/upload/.

mehaase commented 1 year ago

I'm testing this out locally and have a couple other notes:

  1. The payload needs to be form-encoded, not JSON encoded.

  2. The csrftoken and csrfmiddlewaretoken are two different things. To obtain the csrfmiddlewaretoken, you need to fetch the index page and parse the value out of this hidden form element:

Screen Shot 2022-09-16 at 11 00 37 AM

Here's an example to demonstrate. Not recommending you do it this way in production, but it's good enough for testing:

bash-3.2$ csrftoken="34K0NeHfqvEE6Jaj6KYLyWlTMsC0Png7ngKkfeU8V09aBcdRENvlClLcpYgupuux"

bash-3.2$ sessionid="5cq69l6h56vle5wzw5l1msvbal67cbp5"

bash-3.2$ token=$(curl 'http://localhost:8000/' -s \
>   -H "Cookie:csrftoken=$csrftoken;sessionid=$sessionid" \
>   | grep csrfmiddlewaretoken \
>   | cut -d '"' -f 6)

bash-3.2$ curl 'http://localhost:8000/upload/' \
>   -H "Cookie:csrftoken=$csrftoken;sessionid=$sessionid" \
>   -F "csrfmiddlewaretoken=$token"\
>   -F "file=@my_report.txt"
{"message": "File saved for processing.", "job-id": 14, "doc-id": 14}
bash-3.2$

Note that I set two variables at the beginning just to make things easier to read. The third command downloads the main TRAM page, parses out the CSRF token from the form, and saves it into $token. Finally, the fourth command does a form post with the CSRF token obtained in step 3 and reads in a file called my_report.txt.

Let me know if this works for you.

pshurie commented 1 year ago

I'll give that a go.

I have one more question for you:

  1. Another question I have for you has to do with the database. is the data stored in a database that is clustered?The reason why I ask is that we are setting up tram in a kubernetes cluster and wondering if I should use a deployment or a stateful set. If there is no cluster I will just use a deployment with a pod replica set all pointing to the same volume, but if the database is used in a cluster I will need to use a stateful set.

I've been looking at the tram code and see you are using sqlite3 with django so it looks like there is no cluster so I think I want to set it up as a deployment where each pod is accessing the same volume like so:

              tram pod1 --->

nginx --> 1 volume claim ----> volumes (one for /tram/data and the other for /tram/src/tram/staticfiles) tram pod2--->

Does that seem reasonable where all tram instances point to the same volume? (sorry I don't have paint) thanks for your help Mark. I really appreciate it!

On Fri, Sep 16, 2022 at 11:41 AM Mark E. Haase @.***> wrote:

I'm testing this out locally and here a couple other notes:

1.

The payload needs to be form-encoded, not JSON encoded. 2.

The csrftoken and csrfmiddlewaretoken are two different things. To obtain the csrfmiddlewaretoken, you need to fetch the index page and parse the value out of this hidden form element:

[image: Screen Shot 2022-09-16 at 11 00 37 AM] https://user-images.githubusercontent.com/320904/190669930-55d2a164-52af-4ca1-bd6f-68c260799ea2.png

Here's an example to demonstrate. Not recommending you do it this way in production, but it's good enough for testing:

bash-3.2$ csrftoken="34K0NeHfqvEE6Jaj6KYLyWlTMsC0Png7ngKkfeU8V09aBcdRENvlClLcpYgupuux"

bash-3.2$ sessionid="5cq69l6h56vle5wzw5l1msvbal67cbp5"

bash-3.2$ token=$(curl 'http://localhost:8000/' -s > -H "Cookie:csrftoken=$csrftoken;sessionid=$sessionid" > | grep csrfmiddlewaretoken > | cut -d '"' -f 6)

bash-3.2$ curl 'http://localhost:8000/upload/' > -H "Cookie:csrftoken=$csrftoken;sessionid=$sessionid" > -F "csrfmiddlewaretoken=$token"> -F @.***_report.txt" {"message": "File saved for processing.", "job-id": 14, "doc-id": 14} bash-3.2$

Note that I set two variables at the beginning just to make things easier to read. The third command downloads the main TRAM page, parses out the CSRF token from the form, and saves it into $token. Finally, the fourth command does a form post with the CSRF token obtained in step 3 and reads in a file called my_report.txt.

Let me know if this works for you.

— Reply to this email directly, view it on GitHub https://github.com/center-for-threat-informed-defense/tram/issues/183#issuecomment-1249517900, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFSHRMDSZUXTU5S56ZS6JTLV6SIKPANCNFSM6AAAAAAQI3W3OY . You are receiving this because you were mentioned.Message ID: @.***>

mehaase commented 1 year ago

The database is a SQLite file. You are correct in thinking about a volume to share the data between pods, however I don't think it's going to work well in practice. SQLite is not designed to scale up to a lot of concurrent users and requires an exclusive lock anytime a change is written to the database. I'm not sure if the locking mechanism will work in the k8s volume scenario, and even if it does, I expect the resulting performance to be disappointing. At best, concurrent access would be slow, and at worst users would see strange errors or even data corruption. (This is just my best guess--I haven't actually tested out Django+SQLite+K8s in the configuration above.)