Dradis-CE 3.1.0RC2 or 3.6 - Failed Recovery from Backup - Validation Failed - Taggable Can't be Blank

[djsekops]

See http://discuss.dradisframework.org/t/dradis-ce-3-6-failed-project-import-invalid-project-template-format/451

So as part of an effort to upgrade to 3.6 I realised that as of 12th April my 3.1.0RC2 backups are no longer viable. I was trying to export my data from 3.1.0RC2 and import into 3.6. The instance 3.1.0RC2 environment is still workable, but now I am incredibly nervous committing further to using it for another 40 days through a project. But I'm also so far into the project it will be a real battle to switch to something else. Talk about a rock and a hard place.

Here's my extensive notes from the upgrade, backup import attempts and troubleshooting. As you can see there's many days of work spent trying to diagnose this issue:

dradis 3.6 upgrade notes.txt

All backups dated 11th April or earlier restore correctly into Dradis-CE 3.1.0 RC2 or 3.6. The key element to watch for is here:

[23:02:43] New tag detected: !9467bd_critical [23:02:44] New tag detected: !d62728_high [23:02:45] New tag detected: !ff7f0e_medium [23:02:47] New tag detected: !6baed6_low [23:02:47] New tag detected: !2ca02c_info [23:02:49] Wrapping up... [23:02:49] Setting issue_id for evidence

I then tried to manually re-enter data lost since these backups (102 pages copied from current working instance, saving a new backup file to try a node at a time). I then try making subsequent recovery attempts to bring the data into both 3.1.0RC2 or 3.6. All attempts fail with the error: Validation Failed - Taggable Can't be Blank at this point in the import process:

[23:41:26] New tag detected: !9467bd_critical [23:41:26] Validation failed: Taggable can't be blank [23:41:26] Worker process completed.

So i tried immediately exporting the project (in both 3.1.0RC2 & 3.6), reset the database and tried reimporting the same file. All recovery attempts fails with: Validation failed: Taggable can't be blank error.

Each time I've been trying a thor dradis:reset:database / thor dradis:reset:attachments and ALL recoveries fail. I've been trying a full bundle exec thor dradis:reset but also receive the same error mentioned in this thread:

https://github.com/dradis/dradis-ce/issues/76

It appears not only have the backup files become corrupted for some unknown reason, but when trying to recovery from backups, whilst the initial recovery works, all subsequent rework, which is then backed up cannot be imported into Dradis again should another recovery be required. Is seems Dradis does not have a viable backup recovery system.

The 11th April file: The dradis-repository.xml is 3.3MB, the 12th is 1.7MB. All backups files dated 12th April onwards fail.

I could keep using the current working 3.1.0RC2 version with data current as of 21st April, however I now know that backups cannot be recovered.

I need to think long and hard where to go from here, so any guidance to resolve this, much appreciated.

Thanks

[rachkor]

Hi @Kalaratri, I wanted to let you know that the technical team is looking into this. I don't have any more details for you at this point but we will be back as soon as they're available. Thanks for passing this on to us so that we can investigate and resolve this for you!

[djsekops]

Thanks for your support! If you'd like a copy of any of the relevant files let me know how I can get them to you privately.

[dormi]

@Kalaratri if you can send us those backup files (working and broken) that could help to find out what is going on. Also, if it's possible, add for each backup file from what dradis version it was generated and how it was generated (thor?, "export results" ? ...) You could send the files to email@securityroots.com or via https://dradis.slack.com

[djsekops]

Hi

Great, thanks for your assistance.

I've tried to upload the files to slack - too big for email, but I think I need an invitation to register an account?

Both backup files I will send you were generated from 3.1.0RC2. The 11th works fine and I can reimport that, but as I mentioned any rework later saved as a package using the export results > package feature fails to re-import with the "Validation Failed - Taggable Can't be Blank" error in either 3.1.0RC2 / 3.6, so there are really two issues here:

What caused the backup files to become corrupted, presumably on 12th April? I obviously did something to break it!

How can the error be resolved to allow me to get back to a point where I have valid backups of data, whether it's using 3.1.0RC2 / 3.6?

If it helps in any way I have been keeping a diary of the work I've been doing. On the 12th April I was working on the 10.11.1.202 node amongst others and I also did another import from Metasploit

Again, many thanks, look forward to your slack invite.

---

From: Xavi Vila notifications@github.com Sent: Wednesday, April 26, 2017 9:24 AM To: dradis/dradis-ce Cc: Kalaratri; Mention Subject: Re: [dradis/dradis-ce] Dradis-CE 3.1.0RC2 or 3.6 - Failed Recovery from Backup - Validation Failed - Taggable Can't be Blank (#112)

@Kalaratrihttps://github.com/Kalaratri if you can send us those backup files (working and broken) that could help to find out what is going on. Also, if it's possible, add for each backup file from what dradis version it was generated and how it was generated (thor?, "export results" ? ...) You could send the files to email@securityroots.com or via https://dradis.slack.com

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/dradis/dradis-ce/issues/112#issuecomment-297264600, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AY4MEsKhEXDSPWT8T2iyvaxEpV-wFflhks5rzvEmgaJpZM4NFBpf.

[djsekops]

I've signed up for my own slack account and send you an invite to email@securityroots.com

[dormi]

cannot see that invitation yet 😞 Please get your invitation to dradis.slack.com in https://evening-hamlet-4416.herokuapp.com/

[djsekops]

Awesome, joined, which user do I share the backup files with?

[dormi]

@Kalaratri I think this can be closed, a new issue with details on how to reproduce this is available https://github.com/dradis/dradis-ce/issues/116

[djsekops]

Agreed, thanks for your first class support! The bad characters copied and pasted into evidence were indeed breaking the backups