joachimmetz
commented
1 year ago - what format hist.db ?
- is log2timeline.py able to determine the variables used in the artifact definition?
Open erinshore66 opened 1 year ago
joachimmetz
commented
1 year ago 
erinshore66
commented
1 year ago
- what format hist.db ? sqlite -- 20230327
$ pinfo.py hist.db
** Plaso Storage Information *** Filename : hist.db Format version : 20230327 Serialization format : json
> * is log2timeline.py able to determine the variables used in the artifact definition?
I dont know how to answer this. I dont think so. It happens with all %%users.*%% variables
For example: WindowsUserRegistryFilesI dont know how to answer this. I dont think so. It happens with all %%users.*%% variables
what do the debug logs tell you or the user accounts in the hist.db database ?
Also see: https://plaso.readthedocs.io/en/latest/sources/Troubleshooting.html
erinshore66
commented
1 year ago First at all, thank you very much for your help.
what do the debug logs tell you or the user accounts in the hist.db database ?
The hist.db doesnt have any events:
$ pinfo.py hist.db ************************** Plaso Storage Information *************************** Filename : hist.db Format version : 20230327 Serialization format : json -------------------------------------------------------------------------------- *********************************** Sessions *********************************** d962f0e8-627f-44dc-9436-9803034faf74 : 2023-11-04T10:37:28.546754+00:00 -------------------------------------------------------------------------------- ******************************** Event sources ********************************* Total : 0 -------------------------------------------------------------------------------- No events stored. No events labels stored. No warnings stored. No analysis reports stored.I see in the logs lines like these:
2023-11-04 10:37:46,357 [DEBUG] (MainProcess) PID:24890 <artifact_filters> building find spec from path: C:\Users\nromanoff\AppData\Local\Microsoft\Windows\WebCache\WebCacheV*.dat 2023-11-04 10:37:46,357 [WARNING] (MainProcess) PID:24890 <artifact_filters> The path filter must be defined as an absolute path: "C:\Users\nromanoff\AppData\Local\Microsoft\Windows\WebCache\WebCacheV*.dat" 2023-11-04 10:37:46,357 [DEBUG] (MainProcess) PID:24890 <artifact_filters> building find spec from path: C:\Users\rsydow-a\AppData\Local\Microsoft\Windows\WebCache\WebCacheV*.dat 2023-11-04 10:37:46,357 [WARNING] (MainProcess) PID:24890 <artifact_filters> The path filter must be defined as an absolute path: "C:\Users\rsydow-a\AppData\Local\Microsoft\Windows\WebCache\WebCacheV*.dat"I have been testing and if I change in the artifact definitions the %%users.%% variables to the corresponding \Users\\\AppData.... , it works. I have changed InternetExplorerHistory (ONLY InternetExplorerHistory) paths in /usr/share/artifacts/webbrowser.yaml:
Snippet:
....
- '%%users.localappdata%%\Microsoft\Windows\WebCache\WebCacheV*.dat'
To:
- '\Users\*\AppData\Local\Microsoft\Windows\WebCache\WebCacheV*.dat'Then I ran log2timeline again with the BrowserHistory artifact
$ log2timeline.py -d --parsers webhist --artifact_filters BrowserHistory --vss_store=none --storage_file history.db /mnt/hgfs/imgs/base-rd-04-cdrive.E01This way the warnings disappear and I get the Internet Explorer history.
$ pinfo.py history.db
************************** Plaso Storage Information ***************************
Filename : history.db
Format version : 20230327
Serialization format : json
--------------------------------------------------------------------------------
*********************************** Sessions ***********************************
f87fe762-2b36-47a3-824e-f40ee8cc2d32 : 2023-11-04T19:01:55.014016+00:00
--------------------------------------------------------------------------------
******************************** Event sources *********************************
Total : 4
--------------------------------------------------------------------------------
************************* Events generated per parser **************************
Parser (plugin) name : Number of events
--------------------------------------------------------------------------------
msie_webcache : 2995
Total : 2995
--------------------------------------------------------------------------------
No events labels stored.
******************* Extraction warnings generated per parser *******************
Parser (plugin) name : Number of warnings
--------------------------------------------------------------------------------
esedb/msie_webcache : 6
--------------------------------------------------------------------------------
************** Path specifications with most extraction warnings ***************
Number of warnings : Pathspec
--------------------------------------------------------------------------------
3 : type: OS, location: /mnt/hgfs/imgs/base-rd-04-cdrive.E01
: type: EWF
: type: NTFS, location:
\Users\administrator.shieldbase\AppData\Local\Microsoft\Windows\WebCache\WebCacheV01.dat,
MFT attribute: 2, MFT entry: 6867
3 : type: OS, location: /mnt/hgfs/imgs/base-rd-04-cdrive.E01
: type: EWF
: type: NTFS, location:
\Users\nromanoff\AppData\Local\Microsoft\Windows\WebCache\WebCacheV01.dat,
MFT attribute: 2, MFT entry: 12178
--------------------------------------------------------------------------------
No analysis reports stored.Should I change all %%users.*%% variables in the artifact yamls and set absolute paths avoiding these user variables? Or use custom artifacts??
Thank you very much again.
joachimmetz
commented
1 year ago I'll have a closer look when time permits.
erinshore66
commented
9 months ago Hi again! I have been testing the new version to see if this behavior with user variables persisted and I have confirmed that it does.
I have been debugging in the artifact_filters.py file and the _BuildFindSpecsFromFileSourcePath function. with two artifact filters that are a clear example of the observed behavior.
The function is:
def _BuildFindSpecsFromFileSourcePath(
self, source_path, path_separator, environment_variables, user_accounts):
"""Builds find specifications from a file source type.
Args:
source_path (str): file system path defined by the source.
path_separator (str): file system path segment separator.
environment_variables (list[EnvironmentVariableArtifact]):
environment variables.
user_accounts (list[UserAccountArtifact]): user accounts.
Returns:
list[dfvfs.FindSpec]: find specifications for the file source type.
"""
find_specs = []
for path_glob in path_helper.PathHelper.ExpandGlobStars(
source_path, path_separator):
logger.debug('building find spec from path glob: {0:s}'.format(
path_glob))
for path in path_helper.PathHelper.ExpandUsersVariablePath(
path_glob, path_separator, user_accounts):
logger.debug('building find spec from path: {0:s}'.format(path))
if '%' in path:
path = path_helper.PathHelper.ExpandWindowsPath(
path, environment_variables)
logger.debug('building find spec from expanded path: {0:s}'.format(
path))
if not path.startswith(path_separator):
logger.warning((
'The path filter must be defined as an absolute path: '
'"{0:s}"').format(path))
continue
try:
find_spec = dfvfs_file_system_searcher.FindSpec(
case_sensitive=False, location_glob=path,
location_separator=path_separator)
except ValueError as exception:
logger.error((
'Unable to build find specification for path: "{0:s}" with '
'error: {1!s}').format(path, exception))
continue
find_specs.append(find_spec)
return find_specsThe WindowsSystemRegistryFiles filter does everything as expected. System variables like %%environ_systemroot%% are translated correctly and the _BuildFindSpecsFromFileSourcePath function gets the correct path this way:
2024-02-11 12:40:43,680 [DEBUG] (MainProcess) PID:84266 <artifact_filters> building find spec from path glob: %%environ_systemroot%%\System32\config\SYSTEM
2024-02-11 12:40:43,681 [DEBUG] (MainProcess) PID:84266 <artifact_filters> building find spec from path: %%environ_systemroot%%\System32\config\SYSTEM
2024-02-11 12:40:43,681 [DEBUG] (MainProcess) PID:84266 <artifact_filters> building find spec from expanded path: \Windows\System32\config\SYSTEMThe returned path starts with "\" so it continues the execution and gets the right path.
With WindowsUserRegistryFiles and the user variables %%user.*%% the returned path starts with "C:" and not with the path_separator "\" so it enters this if:
if not path.startswith(path_separator):
logger.warning((
'The path filter must be defined as an absolute path: '
'"{0:s}"').format(path))
continueAnd returns the warning we can see in the logs:
2024-02-11 12:44:50,602 [DEBUG] (MainProcess) PID:84295 <artifact_filters> building find spec from path: C:\Users\nromanoff\NTUSER.DAT
2024-02-11 12:44:50,602 [WARNING] (MainProcess) PID:84295 <artifact_filters> The path filter must be defined as an absolute path: "C:\Users\nromanoff\NTUSER.DAT"And is never saved in the variable _findspec because of the continue command in the previous if.
try:
find_spec = dfvfs_file_system_searcher.FindSpec(
case_sensitive=False, location_glob=path,
location_separator=path_separator)Now, we add a new if to remove "C:" string from the returned path
if path.startswith("C:"):
path = path[2:]
logger.debug(('ROBI REMOVES STRING C: '
'"{0:s}"').format(path))This time the path starts with the expected path_separator "\" and the execution continues:
2024-02-11 13:02:28,607 [DEBUG] (MainProcess) PID:84369 <artifact_filters> building find spec from path: C:\Users\nromanoff\NTUSER.DAT
2024-02-11 13:02:28,607 [DEBUG] (MainProcess) PID:84369 <artifact_filters> ROBI REMOVES STRING C: "\Users\nromanoff\NTUSER.DAT"And it works! Let's try the original command that I opened this issue with. If I run the BrowserHistory again it works perfectly:
log2timeline.py -d --parsers webhist --artifact_filters BrowserHistory --vss_store=none --storage_file hist.db /mnt/hgfs/imgs/base-rd-04-cdrive.E01
psort.py -o l2tcsv -w hist.csv hist.dbI get back what I expected:
.....
12/05/2018,14:50:11,UTC,....,WEBHIST,MSIE WebCache container record,Expiration Time,-,BASE-RD-04,URL: Visited: spsql@https://login.live.com/oauth20_authorize.srf?client_id=00...,URL: ........
12/10/2018,01:54:52,UTC,....,WEBHIST,MSIE WebCache container record,Expiration Time,-,BASE-RD-04,URL: Visited: nromanoff@https://login.live.com/login.srf?wa=wsignin1.0&rpsnv=...,URL: Visit..
.....I don't know if this change may have an impact on other Plaso capabilities. I am not a Python expert. I hope this can help. Thank you very much.
Describe the problem:
When I run log2timeline with BrowserHistory artifact I dont get any results nor do I see any error.
To Reproduce:
Plaso Version: Latest: 20230717 OS Version; Ubuntu 22.04.3 LTS (Fresh install) Source data: base-rd-04-cdrive.E01 from FOR508 Installation Method: I installed plaso with the recommendations of the official documentation (add universe; add ppa; and apt install plaso-tools).
Steps to reproduce:
Run log2timeline with BrowserHistory artifact on base-rd-04 image from FOR508 with this command:
Command output:
Log Output:
Then I ran psort in this way and no results found:
The same happens if I run with docker.
Expected Behavior Get browser history from an image