New project directory structure - [merged] #1165

Closed spine-o-bot closed 3 years ago

In GitLab by @PekkaSavolainen on Nov 14, 2019, 17:58

_Merges issue#257_new_project_directorystructure -> dev

Resolves issue #257. Projects are stored into a directory after this. You can still open old .proj files but they will be upgraded to a new style project immediately. There's a new class called ProjectUpgrader that we can hopefully use in the future to upgrade projects automatically when we need to make changes to how projects are saved. There's a new key in all project.json files ('version') that is meant to indicate the project file version.

File paths are now saved to project.json as absolute or relative. If the file is in the project directory, the path is saved as relative. If the file is outside the project directory, the path is saved as absolute. This affects the following:

Tool specification paths
Data Connection references
Data Store database key in url if the database is a file
Exporter file paths

To test:

Start App
App starts without a project even if you had the Open recent project at start up selected.
Go to File -> Open Project. This opens a new dialog where you can either choose an old .proj file to open or later when you have one a Spine Toolbox project directory
Choose an old .proj file to open. Press ok
This forces you to save the old project using the new style immediately by letting you choose a project Directory
A new directory will be created and all project item data is copied to the new directory. Check that all project item data is copied successfully. If you choose a directory that is already a Spine Toolbox project, it will ask if you want to overwrite the existing.
NOTICE Even though data files have been copied to the new project directory, nothing in the project item Properties has been updated. For example, if you have Data Stores that reference .sqlite files in the old project directory, these will still reference the old directory until you manually update them. Also, e.g. Exporter settings may need some updating.
Check something else. e.g. Tool execution.
You can now change the Project name by just going to Settings and typing a new Project name to the appropriate line edit. Check this too. Known issue: This will make an extra entry in File -> Open recent list.
Work directory is no longer a Project setting so this is not saved to project information json (/.spinetoolbox/project.json). The work directory is saved via QSettings just like any other application global setting. Change this and see if it works.
File -> Save Project As... makes a duplicate of the Project into another directory and opens the new one.
Save project, exit app, and check that the previous project is loaded successfully
Create a new Project. As a new step, you need to provide a directory for the new project.

TODO:

[x] Update tests
[x] Do something about absolute paths in project.json files
[x] Reimplement Save Project As

TLDR; Test and give feedback, please.

In GitLab by @PekkaSavolainen on Nov 14, 2019, 17:59

changed the description

In GitLab by @PekkaSavolainen on Nov 14, 2019, 17:59

changed target branch from master to dev

In GitLab by @PekkaSavolainen on Nov 14, 2019, 18:00

In issue #257 description, there's a task:

Remove absolute paths from project save files. Also, the paths should be stored using forward slash ('/'), which is understood by Python in all systems

I'm not sure what is the point in removing the absolute paths. What is the alternative? Relative paths? I can kind of see the point if the project only uses files in the project directory, but what about files that are outside the project directory?

Also, I'm not sure about the second sentence either. The paths are now stored using Python's os.path.abspath(). So for example a data connection reference looks like this. "references": ["C:\\data\\temp\\file.txt"]. What does this look like on Linux? @Manuel?!?

In GitLab by @PekkaSavolainen on Nov 14, 2019, 18:07

added 5 commits

0a041701...881d6ddc - 4 commits from branch dev
9969080a - Merge branch 'dev' into issue#257_new_project_directory_structure

Compare with previous version

In GitLab by @PekkaSavolainen on Nov 14, 2019, 18:17

added 1 commit

f77bf43a - _update_basedir method referenced a non existing work dir location. This is now fixed.

Compare with previous version

In GitLab by @PekkaSavolainen on Nov 14, 2019, 18:19

Note that the work dir is now printed Three times to Event Log when a Tool is executed. The same problem is now in dev.

In GitLab by @PekkaSavolainen on Nov 14, 2019, 18:31

added 1 commit

7ca88f1c - Moved ProjectDirectoryIconProvider class to helpers and removed an unnecessary...

Compare with previous version

In GitLab by @soininen on Nov 15, 2019, 08:17

If I remember correctly the thinking behind the relative paths is that you can move the project folder around in the file system without breaking anything. I think it makes sense to use relative paths for things that are in the project folder. URLs pointing to e.g. external databases or input files should still remain absolute.

Also, following the same thinking the relative paths should be in system-independent representation so you can copy a project from Windosws to OSX, for instance. The sore point here is the external resources (databases, input files) that should be referred to by absolute paths; I don't think it makes much sense to store these in system independent format as the directory structures are wildly different between Windows and *NIX. Perhaps we should just accept the little annoyance that users would need to configure their data connections again if they happen to move the project between different operating systems. I guess that does not happen too often, anyway.

In GitLab by @soininen on Nov 15, 2019, 08:25

if isinstance(info, QFileIconProvider.IconType):?

In GitLab by @soininen on Nov 15, 2019, 08:31

if not answer:?

In GitLab by @soininen on Nov 15, 2019, 08:35

if not answer:?

There are more of these below.

In GitLab by @soininen on Nov 15, 2019, 08:50

Even though data files have been copied to the new project directory, nothing in the project item Properties has been updated. For example, if you have Data Stores that reference .sqlite files in the old project directory, these will still reference the old directory until you manually update them. I'm not sure if we have other properties that should be updated after upgrading a project (Exporter settings maybe?!?)

Exporter has its own export settings files in the project directory that indeed have the same problem as the .sqlite files. This boils down to using absolute paths in the .proj file. I guess users need to update the Exporter settings file path manually. Luckily, there should not be too many of them in the wild at the moment.

In GitLab by @soininen on Nov 15, 2019, 08:51

assigned to @soininen

In GitLab by @PekkaSavolainen on Nov 15, 2019, 09:56

Sure, I can change this but what is the difference?

In GitLab by @PekkaSavolainen on Nov 15, 2019, 10:00

changed this line in version 6 of the diff

In GitLab by @PekkaSavolainen on Nov 15, 2019, 10:00

changed this line in version 6 of the diff

In GitLab by @PekkaSavolainen on Nov 15, 2019, 10:00

added 1 commit

b9e082b7 - Changed all <if answer == "":> to <if not answer:>

Compare with previous version

In GitLab by @PekkaSavolainen on Nov 15, 2019, 10:03

added 2 commits

7d369863 - 1 commit from branch dev
55be15cf - Merge branch 'dev' into issue#257_new_project_directory_structure

Compare with previous version

In GitLab by @PekkaSavolainen on Nov 15, 2019, 10:03

Done

In GitLab by @PekkaSavolainen on Nov 15, 2019, 10:04

Done

In GitLab by @soininen on Nov 15, 2019, 10:04

It takes into account subclasses of QFileIconProvider.IconType and it is also the canonical way of checking the type in Python.

In GitLab by @PekkaSavolainen on Nov 15, 2019, 10:08

changed this line in version 8 of the diff

In GitLab by @PekkaSavolainen on Nov 15, 2019, 10:08

added 1 commit

25a5ac45 - Using isinstance() to check the type of an instance instead of __class__()

Compare with previous version

In GitLab by @soininen on Nov 15, 2019, 10:11

I might be missing something here, but why do I get two directories in the work directory when executing? One of them contains the Tool script and the output files while the other one is just empty.

In GitLab by @soininen on Nov 15, 2019, 10:14

Two questions:

What is the reasoning behind rolling out our own Open project dialog? Are the system's standard dialogs missing some functionality?
Why is there a .spinetoolbox directory in the project directly?

In GitLab by @PekkaSavolainen on Nov 15, 2019, 10:33

Standard QFileDialog does not support opening both files and directories. You can either use a QFileDialog to select a directory or to select a file.
That is according to specifications given by @ererkka in Issue #257 discussion.

In GitLab by @PekkaSavolainen on Nov 15, 2019, 10:36

I'm not sure I follow. I don't think I touched execution at all. Does the same happen in dev?

In GitLab by @soininen on Nov 15, 2019, 10:39

Fair enough. I was afraid this would be the case.
Thanks for the reference! In that view it makes sense to have .spinetoolbox/ indeed.

In GitLab by @soininen on Nov 15, 2019, 10:44

This seems to be present in dev so never mind. I should have checked that first. Sorry for the noise!

In GitLab by @ererkka on Nov 15, 2019, 10:51

After setting up a Tool, I get

[15-11-2019 10:48:36] Tool Tool 1 added to project.
[15-11-2019 10:50:16] Unsupported project filename C:\Users\ERERKKA\Documents\Spine\CS A4\.spinetoolbox\project.json. Extension should be .proj.

In GitLab by @PekkaSavolainen on Nov 15, 2019, 10:55

Should we create some more directories automatically?

All projects now have directories

/.spinetoolbox
/.spinetoolbox/items

and the project info file in

/.spinetoolbox/project.json

In GitLab by @PekkaSavolainen on Nov 15, 2019, 10:55

Sorry I forgot to test that part. Fixing...

In GitLab by @soininen on Nov 15, 2019, 11:00

What do you have in mind? Something like the <project_dir>/my_input_data or <project_dir>/src @ererkka is showing in the issue's comment? Might be nice to have the most commonly needed ones generated automatically, but do we know which these actually are? Would it be better to wait a bit and see how people are filling the project directories?

In GitLab by @ererkka on Nov 15, 2019, 11:06

Work dir setting keeps resetting back to {toolbox install dir}/work at every new opening of the app. It is saved in the project.json file correctly, however, and that does not change.

In GitLab by @ererkka on Nov 15, 2019, 11:08

I also think that work dir should be a global setting by default. One could override that for the project if needed.

In GitLab by @ererkka on Nov 15, 2019, 11:10

For example I cannot share this project now:

"project": {
...
"tool_specifications": [
            "C:\\Users\\ERERKKA\\Documents\\Spine\\CS A4\\hello.json"
        ],
... 
}

I think the path should be stored as just hello.json (relative to the project root).

In GitLab by @ererkka on Nov 15, 2019, 11:13

Btw, if there is only one top level key in the JSON ("project"), we could just drop that level and put the second level keys ("name", "description" etc.) as top-level keys.

In GitLab by @ererkka on Nov 15, 2019, 11:14

Oops, there is "objects"as well. Nevermind...

In GitLab by @PekkaSavolainen on Nov 15, 2019, 11:23

I don't understand. I did make the work dir a global setting. It is not saved to project.json file at all. At least it should not be.

Could you paste the project.json file here?

In GitLab by @PekkaSavolainen on Nov 15, 2019, 11:33

We could wait to see how people use the project directories but it would be easy to add those now...

In GitLab by @soininen on Nov 15, 2019, 12:33

Perhaps we could add some directories then, to encourage users to put their own data into the project dir.

In GitLab by @ererkka on Nov 15, 2019, 12:35

I created this project from scratch.

{
    "project": {
        "name": "CS A4",
        "description": "",
        "work_dir": "C:\\Users\\ERERKKA\\AppData\\Local\\Temp",
        "tool_specifications": [
            "C:\\Users\\ERERKKA\\Documents\\Spine\\CS A4\\hello.json"
        ],
        "connections": [
            [
                false
            ]
        ],
        "scene_y": 23.0,
        "scene_w": 111.0,
        "scene_h": 130.5,
        "scene_x": 118.5
    },
    "objects": {
        "Data Stores": {},
        "Data Connections": {},
        "Tools": {
            "Tool 1": {
                "short name": "tool_1",
                "description": "",
                "x": 174.0,
                "y": 98.0,
                "tool": "Hello",
                "execute_in_work": true
            }
        },
        "Views": {},
        "Data Interfaces": {},
        "Data Exporters": {}
    }
}

In GitLab by @ererkka on Nov 15, 2019, 12:44

After pulling to 25a5ac456, I get

Traceback (most recent call last):
  File "spinetoolbox.py", line 23, in <module>
    sys.exit(main())
  File "C:\Users\ERERKKA\Documents\Python\SpineToolbox\spinetoolbox\main.py", line 48, in main
    window.init_project()
  File "C:\Users\ERERKKA\Documents\Python\SpineToolbox\spinetoolbox\ui_main.py", line 288, in init_project
    if not self.open_project(previous_project, clear_event_log=False):
  File "C:\Users\ERERKKA\Documents\Python\SpineToolbox\spinetoolbox\ui_main.py", line 372, in open_project
    if not self.restore_project(proj_info, load_dir, clear_event_log):
  File "C:\Users\ERERKKA\Documents\Python\SpineToolbox\spinetoolbox\ui_main.py", line 387, in restore_project
    version = project_info["project"]["version"]
KeyError: 'version'

In GitLab by @ererkka on Nov 15, 2019, 12:47

I added the "version" key manually with value 1. Then it works.

In GitLab by @PekkaSavolainen on Nov 15, 2019, 13:23

added 2 commits

cea7a213 - 1 commit from branch dev
0531e96c - Merge branch 'dev' into issue#257_new_project_directory_structure

Compare with previous version

In GitLab by @PekkaSavolainen on Nov 15, 2019, 15:10

added 1 commit

4aaff1d8 - Possibly a fix for a double work dir problem

Compare with previous version

In GitLab by @PekkaSavolainen on Nov 15, 2019, 15:11

added 1 commit

2f6ab6ba - Updated unit tests for the new project directory structure

Compare with previous version

In GitLab by @PekkaSavolainen on Nov 15, 2019, 15:12

added 2 commits

692af126 - 1 commit from branch dev
77b1e729 - Merge branch 'dev' into issue#257_new_project_directory_structure

Compare with previous version

In GitLab by @PekkaSavolainen on Nov 15, 2019, 15:14

marked the task Update tests as completed

In GitLab by @PekkaSavolainen on Nov 15, 2019, 15:19

test_execute_complex_tool_in_work_dir() in TestToolExecution class did not pass on my branch so I had to make a small change to Tool class (commit 4aaff1d8). Did that fix the double work directory problem? I don't know why the same test passes on dev though.

spine-tools / Spine-Toolbox

New project directory structure - [merged] #1165