A contribution to JuliaOptimalTransport?

[SebKrantz]

@devmotion, @zsteve and @davibarreira, I have just published this library to the general package repository, and was wondering if it would be a useful addition to your GitHub organization. Needless to say, the problem of finding the optimal transport network solved by this library using methods from quantitative spatial economics is different from classical optimal transport, but no less relevant.

The paper implementing the model is here, the authors have written a large MATLAB toolbox which is also included in this repo. There are several derivative works using this software, notably.

Graff, T. (2024). Spatial inefficiencies in Africa’s trade network. Journal of Development Economics, 103319. https://tilmangraff.github.io/uploads/papers/Graff2022.pdf

Gorton, Nicole and Ianchovichina, Elena, Trade Networks in Latin America: Spatial Inefficiencies and Optimal Expansions. Available at SSRN: https://ssrn.com/abstract=4041108 or http://dx.doi.org/10.2139/ssrn.4041108

Kreindler, G., Gaduh, A., Graff, T., Hanna, R., & Olken, B. A. (2023). Optimal Public Transportation Networks: Evidence from the World's Largest Bus Rapid Transit System in Jakarta (No. w31369). National Bureau of Economic Research.

And my own recent work

Krantz, Sebastian, Optimal Investments in Africa's Road Network (July 23, 2024). Available at SSRN: https://ssrn.com/abstract=4893771 or http://dx.doi.org/10.2139/ssrn.4893771

[zsteve]

Hi @SebKrantz ,

Apologies for the delayed response, and thanks for making this issue. Your package looks very nice, and I think it could be a relevant addition to the JuliaOptimalTransport org, but I'm happy to hear what @devmotion or @davibarreira have to say also. In terms of the package organisation, it looks like the matlab implementation is currently included as part of the repo. is this required for the Julia implementation at all?

Otherwise I am happy to support its addition. If the goal of this package is to be ready for end-user use (rather than being a research code), perhaps some reorganisation and cleanup may be in order.

[SebKrantz]

Hi @zsteve, thanks for the feedback. So strictly speaking this is a bilingual publication. Fajgelbaum & Schaal (2020) have released a MATLAB library which I here translated to modern Julia. Both are well-designed i.e. not research codes. The MATLAB library is not published elsewhere on GitHub, and had some small bugs which I fixed in this repo (there may be further bugs which users may want to create issues for). It also has a very nice PDF documentation which is helpful for Julia users as well. Thus, I'd like to keep it a bilingual repo. The init.jl file insures that it installs as a Julia Library. As mentioned, it is also already available on the general package repository. But yeah, happy for any reorganization suggestions otherwise, and looking forward to further responses.

[zsteve]

Hi @SebKrantz, I see. Given that the Julia implementation doesn't actually require the MATLAB codes (and vice versa, if I am not mistaken), would it not make more sense to split these into two separate repos? That way, users seeking to install/use the Julia version would not also download the MATLAB files as part of Pkg.add. At the same time, a separate repo for the MATLAB codes would give more visibility for matlab community users (I note that the current repo ends with ".jl", from which it may not be apparent that MATLAB codes are also available).

Happy to hear your thoughts on this, if there are specific considerations you have for keeping them together.

Stephen

[SebKrantz]

Ahh ok, I thought once the library is published to the General Repository calling add OptimalTransportNetworks does not download the full repo but only the required files as pointed to by __init__.jl. So yes, it would be possible to split them into two different repos, and that would make sense if what you say about installation is correct, but they need to remain linked, as the MATLAB documentation is also vital for Julia users to understand the library, in particular the theoretical framework, and if course knowing that this is a translation from MATLAB can help with any bug reports.

[zsteve]

I see. I was also unsure of this so I had a quick check:

(base) [syz@syz-x11g11 PDu3l]$ pwd
/home/syz/.julia/packages/OptimalTransportNetworks/PDu3l
(base) [syz@syz-x11g11 PDu3l]$ ls .
LICENSE  OptimalTransportNetworkToolbox_v.1.0.4b  Project.toml  README.md  __init__.jl  docs  examples  misc  src
(base) [syz@syz-x11g11 PDu3l]$ ls OptimalTransportNetworkToolbox_v.1.0.4b/
Code  Docs  LICENSE.txt
(base) [syz@syz-x11g11 PDu3l]$ ls OptimalTransportNetworkToolbox_v.1.0.4b/Code/
LICENSE.txt             constraints_mobility_cgc.m  objective.m               paper_example03.m                       solve_allocation_by_duality_with_inefficiency.m  solve_allocation_mobility_cgc_inefficiency.m
add_node.m              create_auxdata.m            objective_cgc.m           paper_example04.m                       solve_allocation_cgc.m                           solve_allocation_partial_mobility.m
annealing.m             create_graph.m              objective_custom.m        patchline.m                             solve_allocation_cgc_ADiGator.m                  solve_allocation_partial_mobility_cgc.m
apply_geography.m       example01.m                 objective_duality.m       plot_graph.m                            solve_allocation_custom_ADiGator.m               solve_allocation_primal.m
call_adigator.m         example02.m                 objective_mobility.m      remove_node.m                           solve_allocation_mobility.m
constraints.m           example03.m                 objective_mobility_cgc.m  solve_allocation.m                      solve_allocation_mobility.sav
constraints_cgc.m       example04.m                 optimal_network.m         solve_allocation_ADiGator.m             solve_allocation_mobility_ADiGator.m
constraints_custom.m    find_node.m                 paper_example01.m         solve_allocation_by_duality.m           solve_allocation_mobility_cgc.m
constraints_mobility.m  init_parameters.m           paper_example02.m         solve_allocation_by_duality_ADiGator.m  solve_allocation_mobility_cgc_ADiGator.m
(base) [syz@syz-x11g11 PDu3l]$ 

so indeed it looks like the matlab files are being stored.

[SebKrantz]

Thanks. I have now removed the MATLAB library. Let me know whether the repo is now acceptable to you. I would like to keep the examples and misc folders (with paper materials etc.) if posssible. I'm fairly new to Julia, in R there is a config file called .Rbuildignore to list directories that should not be included in the package building process. I far as I can see Julia does not have anything similar. So I guess my __init__.jl solution is the best way.

[SebKrantz]

I'm wondering: would these directories be ignored if I prefix them with a ., e.g. .misc instead of misc?.

[devmotion]

It doesn't matter which names you use, the Julia package manager will always download and install the contents of the whole repo at the registered commit (but of course not the git history). There's no need for an __init__.jl file, it doesn't help and doesn't affect users in any way. Apart from a few special folders and filenames (e.g., Project.toml, src/, test/, docs/, and ext/) you are free to use whatever folder structure you prefer.

[SebKrantz]

Ok, thanks @devmotion! I've now decided to remove all large files from misc (basically the paper and slides), and instead link to the corresponding MATLAB repo which also has them. I've just downloaded main, the total size is 404kb now. I've also removed __init__.jl, and I'm registering the new version now.

More could be done still (testing etc.), but the library works (I've used it for reasearch), and from my side I'd be ready to migrate the library to your organization if you think this is a valuable addition (also @zsteve, @davibarreira), provided of course that I also get invited to the organization and retain full access to it.

[zsteve]

This looks good to me, happy to add it to the org, I will give the others a few days if there are any remaining concerns.

[SebKrantz]

FYI Just in case you guys want to see a real use example: I've just pushed out a replication package for my paper on optimal investments into the African road network: https://github.com/SebKrantz/OptimalAfricanRoads (required by the World Bank for their policy research working papers)

For example you could run this file (takes about 1h): https://github.com/SebKrantz/OptimalAfricanRoads/blob/main/code/11_GE_simulation_trans_african/optimal_trans_african_networks_largest_pcities.jl

and then this R file https://github.com/SebKrantz/OptimalAfricanRoads/blob/main/code/11_GE_simulation_trans_african/analyze_trans_african_results.R

To generate these plots: https://github.com/SebKrantz/OptimalAfricanRoads/tree/main/figures/transport_network/GE/trans_african

[SebKrantz]

Hi guys, I have been in touch with the authors of the Econometrica paper, and now created an organization at https://github.com/OptimalTransportNetworks and put the software there. We will use this as a Platform to further develop the software. Thus, unless you want to fork it from there, I don't think it can be part of JuliaOptimalTransport anymore. But thanks a lot for your consideration and the feedback on the repo structure.