search

tw4l / brunnhilde

Siegfried-based characterization tool for directories and disk images
MIT License
82 stars 11 forks source link

Brunnhilde.py error: Is the path name current re: line 699 main()? #38

Closed EagleRock1313 closed 5 years ago

EagleRock1313 commented 5 years ago

Brunnhilde.py error: Is the path name current re: line 699 main()?

Downloading CSS and JS files from Github... -1 / unknown

Refer to function main()

line 699

download assets

bootstrap_css_url = 'https://github.com/timothyryanwalsh/brunnhilde/blob/master/assets/css/bootstrap.min.css'
bootstrap_js_url = 'https://github.com/timothyryanwalsh/brunnhilde/blob/master/assets/js/bootstrap.min.js'
jquery_url = 'https://github.com/timothyryanwalsh/brunnhilde/blob/master/assets/js/jquery-3.3.1.slim.min.js'
popper_url = 'https://github.com/timothyryanwalsh/brunnhilde/blob/master/assets/js/popper.min.js'

print("\nDownloading CSS and JS files from Github...")
try:
    wget.download(bootstrap_css_url, os.path.join(css, 'bootstrap.min.css'))
    wget.download(bootstrap_js_url, os.path.join(js, 'bootstrap.min.js'))
    wget.download(jquery_url, os.path.join(js, 'jquery-3.3.1.slim.min.js'))
    wget.download(popper_url, os.path.join(js, 'popper.min.js'))

print("\nDownloads complete.")

tw4l commented 5 years ago

Hi, and thanks for creating an issue! I'm not quite sure that I understand the error you're encountering. The "-1/unknown" that is output to the terminal is from wget, and typically an indicator that the file download is in-progress.

I did a quick check of the JS and CSS URLs and they all seem to be accurate.

Is Brunnhilde getting hung up on this step for you?

EagleRock1313 commented 5 years ago

Hi Tim, Yes, the "-1/unknown" is the error message. It does not seem to be hung up. It is not clear if the download is complete, given the error message.

Is this download portion of the script new? I am running an older version in BC 16.04 and do not see those commands in main()?

Thanks. Sandy

On Fri, 11/2/18, Tim Walsh notifications@github.com wrote:

Subject: Re: [timothyryanwalsh/brunnhilde] Brunnhilde.py error: Is the path name current re: line 699 main()? (#38) To: "timothyryanwalsh/brunnhilde" brunnhilde@noreply.github.com Cc: "EagleRock1313" mliseaglerock1313-web@yahoo.com, "Author" author@noreply.github.com Date: Friday, November 2, 2018, 5:10 AM

Hi, and thanks for creating an issue! I'm not quite sure that I understand the error you're encountering. The "-1/unknown" that is output to the terminal is from wget, and typically an indicator that the file download is in-progress. I did a quick check of the JS and CSS URLs and they all seem to be accurate. Is Brunnhilde getting hung up on this step for you?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

stefanabreitwieser commented 5 years ago

Just a note to say that we had this error message since the last update too. It doesn't appear to have any impact on the final results(??)

Thanks Tim!

tw4l commented 5 years ago

Thank you, Sandy and Stefana!

Downloading the CSS and JavaScript files and saving them locally along the report is a new feature, because reports generated by previous versions of Brunnhilde were linking out to externally-hosted CSS and JS files, which was (ironically) a digital preservation risk for the outputs of a digital preservation utility. Less than ideal.

The behavior introduced in Brunnhilde 1.7.2 instead downloads these files from this Github repository each time and saves them in an "assets" directory in the Brunnhilde outputs, so that the HTML report can access them locally. The current implementation is less than ideal because:

  1. It requires an internet connection to run Brunnhilde, as there is not yet an option to cache these files somewhere on the local computer and just use those moving forward. (This is trickier than it sounds to implement, as pip installs Brunnhilde in different locations depending on which operating system you're running and how Python and pip are installed)

  2. The files are being downloaded by the Python wget library, which is what is outputting the "-1/unknown" to the terminal. This is actually normal behavior - aka not a bug - but it sure looks like one, as this issue and both of your comments attest.

To fix this for a future release, I propose:

  1. To introduce a new argument (or two) allowing users to save the CSS/JS assets to a local directory of their choosing, and then in subsequent runs of Brunnhilde to copy the assets from that directory rather than downloading them from Github every time.

  2. To replace that "-1/unknown" terminal message with more user-friendly status indicators like a typical progress bar or at least some variant of "Download in progress..."

What do you think?

EagleRock1313 commented 5 years ago

Hi Tim, I would agree that active internet needed to run Brunnhilde is not an ideal workflow. I would choose option 1 based on that, but have 3 additional thoughts:

1) Could permissions issues on a chosen directory be a problem? Not all users will factor permissions into a decision. 2) How would updates or changes be handled, if needed? 3) What about size? I am sure these files are not too large (more than a few hundred MB). Our current custom dual boot configuration has a limited partition size of 250 GB, so "choosing" a location must factor in the total size of all files. As an example, APT alerts you about how much additional disk space would be required, and if you wish to proceed with installation. Might be helpful info on any UI.

Certainly option 2, replacing the "-1/unknown" message would be a quick workaround for now. Maybe in a next version, introduce the option to save the files locally. We are building a new BitCurator 2.0.6 environment on a minimal install of Ubuntu 18.04 LTS.
We just started the testing the BC environment, and have more testing to do. Just background info for you on our config.

I am interested in any additional thoughts you or Stefana might have.

Best Regards, Sandy

On Fri, 11/2/18, Tim Walsh notifications@github.com wrote:

Subject: Re: [timothyryanwalsh/brunnhilde] Brunnhilde.py error: Is the path name current re: line 699 main()? (#38) To: "timothyryanwalsh/brunnhilde" brunnhilde@noreply.github.com Cc: "EagleRock1313" mliseaglerock1313-web@yahoo.com, "Author" author@noreply.github.com Date: Friday, November 2, 2018, 6:54 AM

Thank you, Sandy and Stefana! Downloading the CSS and JavaScript files and saving them locally along the report is a new feature, because reports generated by previous versions of Brunnhilde were linking out to externally-hosted CSS and JS files, which was (ironically) a digital preservation risk for the outputs of a digital preservation utility. Less than ideal. The behavior introduced in Brunnhilde 1.7.2 instead downloads these files from this Github repository each time and saves them in an "assets" directory in the Brunnhilde outputs, so that the HTML report can access them locally. The current implementation is less than ideal because:

It requires an internet connection to run Brunnhilde, as there is not yet an option to cache these files somewhere on the local computer and just use those moving forward. (This is trickier than it sounds to implement, as pip installs Brunnhilde in different locations depending on which operating system you're running and how Python and pip are installed)

The files are being downloaded by the Python wget library, which is what is outputting the "-1/unknown" to the terminal. This is actually normal behavior - aka not a bug - but it sure looks like one, as this issue and both of your comments attest.

To fix this for a future release, I propose:

To introduce a new argument (or two) allowing users to save the CSS/JS assets to a local directory of their choosing, and then in subsequent runs of Brunnhilde to copy the assets from that directory rather than downloading them from Github every time.

To replace that "-1/unknown" terminal message with more user-friendly status indicators like a typical progress bar or at least some variant of "Download in progress..."

What do you think?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

tw4l commented 5 years ago

Hi Sandy,

The total size of the four CSS and JavaScript files is 277 KB (they are all minified text files, so they don't take much space).

I think a workaround to the permissions and updating issue would be to make the arguments explicit. Something like:

[First time] brunnhilde.py --save-assets /path/to/local/directory

[Subsequent runs] brunnhilde.py --load-assets /path/to/local/directory

And if a user does not use the --load-assets flag, Brunnhilde will just download them from Github as it currently does in 1.7.2.

Thoughts?

EagleRock1313 commented 5 years ago

Hi Tim,

Sure, that option would work. We could script something that has a "default" set of options for Brunnhilde, and would include the --load-assets option each time, after assets have been saved. I would be interested to see the changes, as last year I wanted to add the bulk_extractor regex text feature to the brunnhilde.py parser options. Being part-time, I have not had a chance to get to this project.

Thanks for your time on this small issue Tim. I will keep my eyes open for the update.

Best, Sandy

On Fri, 11/2/18, Tim Walsh notifications@github.com wrote:

Subject: Re: [timothyryanwalsh/brunnhilde] Brunnhilde.py error: Is the path name current re: line 699 main()? (#38) To: "timothyryanwalsh/brunnhilde" brunnhilde@noreply.github.com Cc: "EagleRock1313" mliseaglerock1313-web@yahoo.com, "Author" author@noreply.github.com Date: Friday, November 2, 2018, 8:03 AM

Hi Sandy, The total size of the four CSS and JavaScript files is 277 KB (they are all minified text files, so they don't take much space). I think a workaround to the permissions and updating issue would be to make the arguments explicit. Something like: [First time]

brunnhilde.py --save-assets /path/to/local/directory [Subsequent runs]

brunnhilde.py --load-assets /path/to/local/directory And if a user does not use the --load-assets flag, Brunnhilde will just download them from Github as it currently does in 1.7.2. Thoughts?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

tw4l commented 5 years ago

Hi @EagleRock1313 and @stefanabreitwieser!

I believe this issue is fixed (at least for now) in release 1.8.0. I've also updated the package on PyPI, so you should be able to update with sudo pip install brunnhilde --upgrade.

Let me know how this works out for you and if the documentation is clear, and as always I'm happy for suggestions, criticisms, and pull requests :)

tw4l commented 5 years ago

(I'm closing the issue to help me triage outstanding issues, but please feel free to re-open and comment here if you have feedback!)

EagleRock1313 commented 5 years ago

Thanks Tim! Sandy

On Thu, 11/8/18, Tim Walsh notifications@github.com wrote:

Subject: Re: [timothyryanwalsh/brunnhilde] Brunnhilde.py error: Is the path name current re: line 699 main()? (#38) To: "timothyryanwalsh/brunnhilde" brunnhilde@noreply.github.com Cc: "EagleRock1313" mliseaglerock1313-web@yahoo.com, "Mention" mention@noreply.github.com Date: Thursday, November 8, 2018, 6:03 PM

Hi @EagleRock1313 and @stefanabreitwieser! I believe this issue is fixed (at least for now) in release 1.8.0. I've also updated the package on PyPI, so you should be able to update with sudo pip install brunnhilde --upgrade. Let me know how this works out for you and if the documentation is clear, and as always I'm happy for suggestions, criticisms, and pull requests :)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

schmidtster commented 5 years ago

I am trying to implement the solution as applied through this thread and README, saving the JS and CSS assets (--save_assets) to a local directory and referencing those when loading them (--load_assets). However, we are having trouble constructing the argument and outputting the results to the metadata directory, as listed in the guidelines.

We've tried multiple iterations of the following:

brunnhilde.py -b -d --hash MD5 -z /media/sf_.../...E01 --save_assets /media/sf_TEST-IMAGES/GK90/ metadata

brunnhilde.py --save_assets -b -d --hash MD5 -z /media/sf_.../...E01 /media/sf_TEST-IMAGES/GK90/ metadata

brunnhilde.py --load_assets -b -d --hash MD5 -z /media/sf_.../...E01 /media/sf_TEST-IMAGES/GK90/ > metadata

The last two seemed to pass the arguments, up to the metadata directory. However, the error message either says the directory already exists or no such directory exists. Both errors cancel the operation, preventing any of the checks/reports to be put in the metadata directory. Our general problem is that we want to download the assets and then copy them into an offline environment, reference them in the command line Brunnhilde, then (importantly) try running the Brunnhilde GUI and hope that it also references the local assets. Any help as to the specifics on how to run the command line would be greatly appreciated!

tw4l commented 5 years ago

Hi @schmidtster, thank you for commenting. A few notes:

The --save_assets <directory_name> argument must be made before the 3 positional arguments (source, destination, and basename), which explains why the first iteration was giving you an error.

The <directory_name> must be replaced by the path to a local folder. When I do this on my Macbook, the terminal output I get specifies the --load_assets command to use moving forward. E.g.:

Tims-MBP:Desktop twalsh$ brunnhilde.py --save_assets save_test /Users/twalsh/PII_test_data/ . out

Downloading CSS and JS files from Github...

Downloads complete.

Brunnhilde assets saved locally. To use these in subsequent runs, use this argument: --load_assets "/Users/twalsh/Desktop/save_test"

Can you try doing this on your machine and copy the terminal output to this issue (or email it to me at timothyryanwalsh [at] gmail [dot] com)?

Finally, I'm realizing that I don't think the GUI has been updated to account for these new arguments, since this approach has been something of a stopgap before I have time to work on Brunnhilde 2, which should resolve this issue of the CSS/JS dependencies in a cleaner way. If being able to use this functionality with the GUI is an immediate need for you, I should be able to make those updates in the coming weeks.

schmidtster commented 5 years ago

Thank you for your help! I redid the arguments with --save_assets and instead of trying to pass source, destination, and basename, I just downloaded it to a local folder, as you suggested. Luckily that time I didn't get any errors.

After that, I played around with --load_assets command. The errors I was receiving were insufficient arguments or file/directory already existing. I will see about recreating the error and editing this comment to insert the errors. However, I did get the following argument to work:

brunnhilde.py --load_assets /home/bcadmin/Desktop/Forensics and Reporting/LOAD_ASSETS -b -d --hash MD5 -z /home/bcadmin/Desktop/GK10/image/GK10.E01 /home/bcadmin/Desktop/GK10 metadata

I renamed the assets folder to LOAD_ASSETS and was able to access them through BitCurator's Forensics and Reporting folder. I should have mentioned I am running this on a BitCurator instance.

In order to update the GUI, I took the liberty to insert the following lines into the main.py program on line 149:

# load local assets
self.process_list.append('--load_assets')
self.process_list.append('/home/bcadmin/Desktop/Forensics and Reporting/LOAD_ASSETS')

I put it right after appending brunnhilde.py to process_list. I was thinking the order the arguments are made in the list affect the GUI's construction of the command line argument, though this may not be the case, in which it wouldn't matter where we put this line. This solution only works for my case, because I know where the assets will be, but it wouldn't be universally acceptable.

If you could update the GUI to include this specific case, that would be great but isn't an immediate need. It will be a need later because I am creating documentation detailing the setup process for installing Brunnhilde GUI on BitCurator in an offline environment. Not having to modify the main.py file for every individual instance of the environment would be really helpful.

tw4l commented 5 years ago

Glad to hear this worked out for you! Closing this issue again for now, as I plan to address this issue as part of the Brunnhilde 2 rewrite.

tw4l commented 4 years ago

Hi @EagleRock1313 @schmidtster. I thought you might like to know that I just published Brunnhilde 1.9.0, which removes the Bootstrap dependency from the HTML report and so no longer requires an internet connection or saving/loading assets locally. You can read more in the release notes: https://github.com/tw4l/brunnhilde/releases/tag/v1.9.0

EagleRock1313 commented 4 years ago

Thank you. On Friday, July 31, 2020, 05:49:56 PM PDT, Tessa Walsh notifications@github.com wrote:

Hi @EagleRock1313 @schmidtster. I thought you might like to know that I just published Brunnhilde 1.9.0, which removes the Bootstrap dependency from the HTML report and so no longer requires an internet connection or saving/loading assets locally. You can read more in the release notes: https://github.com/tw4l/brunnhilde/releases/tag/v1.9.0

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.